Parallelization calculation task process planning platform

Abstract

The invention provides a parallelization calculation task process planning platform, which comprises a task planning process design unit and a parallelization calculation task management unit, and is characterized in that the task planning process unit comprises a dynamic operator, a task triggering module, a task instance management module and a task process management module; the task planning process unit is started through a design process, a user edits a task process on a front-end page, the task process is represented as a node representing an algorithm operator, each node comprises an execution parameter, a plurality of nodes are displayed through a directed acyclic graph, and an execution sequence is represented through an arrow relationship; the task process management module executes dynamic operators according to the sequence defined by the process by monitoring a task process starting event, changes the state of nodes in a task flow diagram DAG according to the execution result of the dynamic operators, and changes the state of subsequent nodes into successful execution when the algorithm is successfully executed; and when algorithm execution fails, changing a subsequent node state into execution failure.

Description

Technical Field

[0001] The present invention belongs to the field of computing process planning, and in particular relates to a parallel computing task process planning platform. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the complexity and scale of computing tasks are increasing. Traditional single-machine computing cannot meet the needs of efficient and fast processing of massive data. Therefore, parallel computing has gradually attracted widespread attention as an effective solution. Parallel computing can significantly improve computing efficiency and resource utilization by decomposing tasks into multiple subtasks and executing them simultaneously on multiple processing units. The parallel computing task flow planning platform came into being, aiming to provide an efficient and flexible way to manage and schedule parallel computing tasks.

[0003] Current technologies still have some significant shortcomings in practical applications, which limits their widespread application and development. First, existing parallel computing platforms usually face the complexity of task division and scheduling. In the process of large-scale data processing, how to reasonably divide tasks into multiple subtasks and effectively distribute them to different processing units is a challenging task. Many existing platforms tend to rely on static rules when dividing tasks, which makes it difficult to dynamically adjust according to real-time computing status and resource usage. This static division may lead to idle resources or delays in computing tasks, thereby affecting the overall computing efficiency.

[0004] The system lacks scalability and flexibility. In some application scenarios, the scale and type of computing tasks may change, and the architecture of existing platforms often cannot adapt to new requirements quickly. For example, when new data types need to be processed or new computing nodes need to be added, the existing system may require cumbersome configuration and adjustment, resulting in insufficient system flexibility. At the same time, many parallel computing platforms also have deficiencies in resource management, and are unable to effectively monitor and manage the use of computing resources, which in turn affects the rational allocation and utilization efficiency of resources.

[0005] Existing technologies are also relatively weak in terms of fault tolerance and fault recovery. In large-scale parallel computing, the failure of a single node may cause the failure of the entire task. Traditional platforms often lack effective fault tolerance mechanisms and find it difficult to achieve automatic recovery of tasks. This lack of fault tolerance increases the risk of computing tasks and may lead to data loss and unreliable computing results. Challenges in user experience and ease of use cannot be ignored. Many parallel computing platforms have complex operating interfaces and steep learning curves, making it difficult for non-professional users to get started quickly. In practical applications, users often need to have certain programming and computer knowledge to effectively use these platforms, which limits their scope of application to a certain extent. In short, although the parallel computing task process planning platform is of great significance in improving computing efficiency and resource utilization, the current technical defects still need to be focused on and resolved to promote its application and development in a wider range of fields. Summary of the invention

[0006] The present invention proposes a parallel computing task process planning platform, which allows users to customize and expand algorithm operators through dynamic operator design, thereby enhancing the flexibility and adaptability of the system. This enables users to design suitable computing tasks according to specific needs and meet diverse business needs.

[0007] The technical solution of the present invention is implemented as follows: a parallel computing task process planning platform, including a task planning process design unit and a parallel computing task management unit, the task planning process unit includes a dynamic operator, a task trigger module, a task instance management module and a task process management module; the task planning process unit is started by designing a process, and a user edits the task process on a front-end page, the task process is represented as a node representing an algorithm operator, each node includes execution parameters, multiple nodes are displayed through a directed acyclic graph, and the execution order is represented by an arrow relationship; the task process management module executes the dynamic operator in the order defined by the process by monitoring the task process startup event, and changes the state of the node in the task process diagram DAG diagram according to the execution result of the dynamic operator, when the algorithm is executed successfully, the subsequent node state is changed to execution success; when the algorithm fails to execute, the subsequent node state is changed to execution failure; the task instance management module includes task instance creation, task instance information query and task instance deletion functions.

[0008] In today's data-driven era, the complexity and scale of computing tasks are increasing, and traditional serial computing methods can no longer meet the needs of efficiently processing large-scale data. Therefore, parallel computing has become an effective solution. However, existing parallel computing platforms still have many technical difficulties in task planning and management. The flexibility and visual design of task planning are insufficient. On traditional platforms, users usually need to define and manage task processes through code or complex command line interfaces, which not only increases the difficulty of use, but also reduces user work efficiency. This platform introduces a front-end page editing function, which enables users to intuitively design task processes through a graphical interface, lowers the threshold for use, and improves user experience.

[0009] In terms of task status management and dynamic adjustment, many existing platforms lack real-time feedback mechanisms, and the status changes of task execution are often not reflected in the user interface in a timely manner. This makes it difficult for users to grasp the real-time progress of tasks during task execution. Through the task flow management module, the system can monitor task start events in real time, and automatically update the node status in the task flow chart according to the execution results of dynamic operators, ensuring that users can get feedback on task execution in the first place. Existing parallel computing platforms have deficiencies in the management of task instances, and often can only provide limited functions, such as simple creation and deletion. This makes users face many inconveniences when managing complex tasks. Through the task instance management module, this platform provides functions such as task instance creation, information query and deletion, allowing users to manage multiple task instances more flexibly, significantly improving the efficiency of task management.

[0010] Compared with existing technologies, this parallel computing task process planning platform has significant advantages in many aspects. First, in terms of user interface design, traditional platforms often rely on text or command lines to define tasks, which has a high threshold for use for non-professional users. However, this platform uses graphical task process design to make task creation and management more intuitive. Users can easily build complex task processes by dragging and connecting nodes. This visual design not only improves user work efficiency, but also makes it easier to understand and maintain tasks.

[0011] In terms of dynamic task management, the platform introduces a real-time feedback mechanism that can dynamically update the node status in the task flow chart according to the different status of task execution. This design allows users to clearly understand the execution status of each algorithm operator, discover problems in a timely manner and take measures, avoiding decision-making errors caused by information lag. However, existing technologies can often only provide static task execution status and lack real-time performance, making it difficult for users to grasp the overall progress when handling complex tasks.

[0012] In terms of task instance management, the existing platforms have relatively simple management functions for task instances, which can only simply create and delete task instances, lacking flexibility and scalability. The task instance management module of this platform provides more comprehensive functions, including task instance creation, information query and deletion, etc. Users can easily manage multiple task instances. This flexible management capability is particularly suitable for scenarios that need to process a large number of parallel tasks, improving the availability and efficiency of the system.

[0013] In terms of supporting algorithm operators, the platform allows users to customize dynamic operators, which enhances the scalability of the system. Users can design and implement new algorithm operators according to specific needs, so that the system can adapt to different computing tasks and business scenarios. However, existing parallel computing platforms are usually less flexible in this regard, and users are often restricted by fixed algorithm libraries, making it difficult to make personalized adjustments and optimizations.

[0014] As a preferred implementation, the template submodule built into the dynamic operator includes two functions: operator management and task flow chart management based on operator combination sorting, wherein operator management includes querying built-in operators, querying custom operators, creating custom operators, modifying custom operators, and deleting custom operators;

[0015] Built-in operators are operators built into the system, and custom operators are operators created by users. Built-in operators correspond to static operators in the system and are automatically loaded into the system when the system starts; custom operators correspond to dynamic operators in the system and are created, modified, and deleted by users and updated to the system in real time.

[0016] As a preferred implementation, when the task trigger module receives a request, the user enters the parameters required for task triggering in the front-end interface, creates a task instance in the queue state, and then executes the algorithm. In the new process allocated by the task scheduling process, based on the operator name passed by the task flow management submodule, the algorithm execution entry function module path associated with the operator is obtained, and through parallel computing task management, the algorithm execution entry function module path and request parameters are passed in for execution.

[0017] As a preferred implementation, the parallel computing task management unit includes four functions: operator encapsulation, operator execution, and node and plug-in management. The operator encapsulation encapsulates the user-written code into an executable operator and installs it in a distributed environment, while simultaneously supporting RPC remote calls; the operator encapsulation includes two parts: code templates and distributed environment configuration, wherein the code template part provides editable code, operator information editing, request parameter configuration, and RPC parameter configuration functions; wherein the distributed environment configuration is to install the third-party Python library required for the operator in a distributed environment, and to install the relevant library through plug-in installation, and after the installation is completed, the operating environment of the operator is met.

[0018] As a preferred implementation, the operator execution part includes two parts: distributed execution and stand-alone execution. The distributed execution part includes two parts: client and server. The client calls the server to execute the operator and returns the result of the operator execution through the RPC protocol. The client part includes RPC call, operator execution log and execution result query function. RPC call is to call the operator of the server. The operator execution log records various logs generated by the operator call. The execution result query is to query the result of the operator execution. The server operator execution includes operator loading, memory management, and execution result saving functions. Then the operator is loaded and the operator information is loaded into the memory. The memory management monitors the memory. Release the memory or save it to disk, save the execution result of the server and save it to the data access plug-in. The stand-alone execution includes two parts: the client and the server. The client includes RPC call, operator information query, execution result query, and operator execution log function. The RPC call is to call the server to execute the operator function, and the operator information is queried in real time. The execution result query is to query the result of the current operator execution. The operator execution log records the log generated by the operator call. The server-side operator execution imports the corresponding execution entry module of the operator through the class loading function, calls the entry module, and passes in parameters to execute the operator, and returns the execution result to the client.

[0019] After adopting the above technical solution, the beneficial effects of the present invention are as follows: the design and implementation of this parallel computing task process planning platform has brought significant beneficial effects. The improvement in user experience is the most obvious. Through the simplified graphical interface, users can get started more quickly, reducing learning costs and usage barriers. This ease of use enables more users to participate in the design and management of parallel computing tasks, promoting teamwork and information sharing. The introduction of a real-time feedback mechanism significantly improves the transparency and controllability of task execution. Users can keep track of the execution status of tasks at any time and respond to possible problems in a timely manner, thereby improving the success rate and execution efficiency of tasks. By dynamically updating the task status, users can quickly make adjustments, optimize resource allocation, and improve overall computing performance.

[0020] The comprehensive functions of the task instance management module enable users to manage multiple task instances more efficiently. In practical applications, users often need to handle multiple parallel computing tasks at the same time. Good task instance management capabilities can significantly improve work flexibility and efficiency. This efficient management method is particularly effective in large-scale data processing and complex task scheduling, and can effectively reduce management costs. The ability of users to customize algorithm operators greatly enhances the adaptability and scalability of the system. Users can flexibly design and adjust algorithms according to business needs, allowing the system to quickly respond to market changes and technological advances. This flexibility enables the platform to remain competitive in a changing business environment and provide users with more personalized and efficient computing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 This is a system block diagram of the platform of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Example:

[0025] like Figure 1As shown, a parallel computing task process planning platform includes a task planning process design unit and a parallel computing task management unit, wherein the task planning process unit includes a dynamic operator, a task trigger module, a task instance management module and a task process management module; the task planning process unit is started by designing a process, and the user edits the task process on the front-end page, and the task process is represented as a node representing an algorithm operator, each node includes execution parameters, and multiple nodes are displayed through a directed acyclic graph, and the execution order is represented by arrow relationships; the task process management module executes dynamic operators in the order defined by the process by monitoring task process start events, and changes the states of nodes in the task process diagram DAG diagram according to the execution results of the dynamic operators. When the algorithm is executed successfully, the states of subsequent nodes are changed to execution success; when the algorithm fails to be executed, the states of subsequent nodes are changed to execution failure; the task instance management module includes task instance creation, task instance information query and task instance deletion functions.

[0026] The working principle and workflow of this parallel computing task process planning platform mainly revolve around the dynamic planning and execution management of tasks, aiming to improve the execution efficiency and flexibility of computing tasks. The following is a detailed description of the specific operation steps and workflow of the platform.

[0027] Users design task flows using the task planning process design unit on the front-end page. Users can create a new task flow through an intuitive graphical interface. The task flow consists of multiple nodes, each of which represents an algorithm operator. Users can combine different algorithm operator nodes together to form a directed acyclic graph (DAG) through drag and drop and connection functions. In each node, users can set corresponding execution parameters, such as input data, algorithm type, computing resources, etc. This design enables users to build complex computing task flows in a visual way without having to deeply understand the underlying code or implementation details.

[0028] Once the task flow design is completed, the user can start the flow through the front-end interface. At this time, the task trigger module will listen to the flow start event and pass the task flow to the task flow management module. After receiving the start event, the task flow management module will execute each dynamic operator one by one according to the order of the flow definition. The execution status of each dynamic operator will be recorded, and the system will update the status of the corresponding node in the DAG diagram in real time according to the execution result.

[0029] During the execution process, if a dynamic operator is successfully executed, the task flow management module will change the status of its subsequent nodes to "execution successful". The system will continue to execute the next dynamic operator until all nodes are processed. If an operator fails to execute during the execution process, the system will immediately update the status of the subsequent nodes to "execution failed" and stop the execution process. This mechanism ensures the efficiency and reliability of the task flow and can quickly respond to problems in execution.

[0030] During the task execution process, the task instance management module also plays a role. This module allows users to manage task instances, including creation, query and deletion functions. When a user starts a new task process, the system will automatically create a corresponding task instance and record the relevant information of the instance in the task instance management module. This information may include the ID, creation time, execution status, related parameters, etc. of the task instance. Users can query the information of the task instance at any time to understand the execution progress and results of the task. If the user needs to process the completed task instance, such as deleting a task instance that is no longer needed, the task instance management module provides the corresponding deletion function. The user only needs to select the task instance to be deleted on the interface and confirm the operation, and the system will remove it from the record.

[0031] In order to enhance the flexibility of the system and user experience, the platform may also provide some additional functions. For example, users can save and version the designed task flow for subsequent modification and optimization. In addition, the system can also support the parallel execution of tasks, that is, when resource conditions are met, multiple task flows are executed simultaneously to improve the overall computing efficiency. The design of the entire workflow makes the management of parallel computing tasks more efficient and intuitive. Users do not need to have a deep programming background to complete complex computing task planning through simple operations. This method not only improves the utilization efficiency of computing resources, but also effectively shortens the execution time of tasks to meet modern computing needs.

[0032] Based on this parallel computing task process planning platform, users can easily design, manage and execute complex computing tasks, monitor the execution status of tasks in real time, and quickly respond to potential problems, thereby improving overall efficiency and reliability in the field of data processing and computing.

[0033] The template submodule built into the dynamic operator includes two functions: operator management and task flow chart management based on operator combination sorting, wherein operator management includes querying built-in operators, querying custom operators, creating custom operators, modifying custom operators and deleting custom operators; wherein built-in operators are operators built into the system, and custom operators are operators created by users. Built-in operators correspond to static operators in the system and are automatically loaded into the system when the system starts; custom operators correspond to dynamic operators in the system and are created, modified, deleted by users and updated to the system in real time.

[0034] In the prior art, the management of dynamic operators is often relatively simple, lacking flexibility and ease of use. Users may face problems such as cumbersome operations and single functions during use. However, this technical solution provides a more comprehensive and flexible operator management function through a built-in template submodule, including operator management and task flow chart management based on operator combination sorting.

[0035] Specifically, the operator management function of this technical solution includes querying built-in operators, querying custom operators, creating custom operators, modifying custom operators, and deleting custom operators. This diverse management function enables users to easily access all available operators in the system, whether they are built-in operators preset by the system or user-defined operators. In addition, built-in operators correspond to static operators in the system, which are automatically loaded when the system starts, ensuring that users can quickly use these core functions at any time. This design significantly improves the system startup efficiency and user experience.

[0036] On the other hand, custom operators correspond to dynamic operators in the system. Users can create, modify, and delete them according to their own needs, and these operations will be updated in real time to the system. This real-time update mechanism enables users to flexibly adjust and optimize their workflows to adapt to rapidly changing needs. However, existing technologies are usually insufficient in this regard. Users often need a long time or complex operation process to manage custom operators, which limits their flexibility and adaptability.

[0037] This technical solution also introduces a task flow chart management function based on operator combination sorting, which enables users to more intuitively manage and arrange the order of operator use and optimize workflow settings. Through this visual management method, users can easily understand and adjust the execution order of tasks, thereby improving overall work efficiency. This is usually lacking in the prior art, where users can only manage through text or command lines, making it difficult to form a clear logical structure.

[0038] When the task trigger module receives a request, the user enters the parameters required for task triggering in the front-end interface, creates a task instance in the queue state, and then executes the algorithm. In the new process allocated by the task scheduling process, based on the operator name passed by the task flow management submodule, the algorithm execution entry function module path associated with the operator is obtained, and through parallel computing task management, the algorithm execution entry function module path and request parameters are passed in for execution.

[0039] In the prior art, the triggering and execution of tasks are often linear and static. When submitting a task request, the user may need to predefine all parameters and execution paths, lacking the ability to flexibly adjust according to specific needs at runtime. This approach not only reduces the flexibility of the system, but may also lead to waste of resources and reduced efficiency. However, through the design of the task trigger module, this technical solution allows users to dynamically input the parameters required for task triggering in the front-end interface and create task instances in the queue state in real time. This dynamic creation mechanism significantly improves the flexibility and responsiveness of the system.

[0040] Specifically, when the user enters the parameters required for task triggering in the front-end interface, the system can immediately create a task instance in a queue state without defining all execution paths and parameters before the system starts. This design allows users to quickly generate tasks according to real-time needs, avoiding cumbersome manual configuration and potential errors. In addition, during the task execution process, the system assigns the task scheduling process to a new process. This multi-process management method ensures the efficient execution of tasks and can make full use of the system's computing resources. In terms of obtaining the algorithm execution entry function module path associated with the operator, the technical solution dynamically obtains the corresponding execution path through the operator name passed by the task flow management submodule. This dynamic path acquisition method greatly improves the flexibility and scalability of the system compared to the static path configuration method commonly used in the prior art. Users can flexibly select and configure operators according to the needs of specific tasks, thereby optimizing the execution process of tasks.

[0041] Through parallel computing task management, the system can pass in the algorithm execution entry function module path and request parameters for execution. This parallel computing capability not only improves the efficiency of task processing, but also reduces the time delay of task execution, allowing the system to remain efficient and stable when processing a large number of tasks. Existing technologies often lack effective parallel processing mechanisms in this regard, resulting in reduced system performance under high load conditions.

[0042] The parallel computing task management unit includes four functions: operator encapsulation, operator execution, node and plug-in management. The operator encapsulation encapsulates the user-written code into an executable operator and installs it in a distributed environment, and simultaneously supports RPC remote calls; the operator encapsulation includes two parts: code template and distributed environment configuration, wherein the code template part provides editable code, operator information editing, request parameter configuration, and RPC parameter configuration functions; wherein the distributed environment configuration is to install the third-party Python library required for the operator in a distributed environment, and install the relevant library through plug-in installation. After the installation is completed, the operating environment of the operator is met.

[0043] The operator execution part includes two parts: distributed execution and single-machine execution. The distributed execution part includes two parts: client and server. The client calls the server to execute the operator and returns the result of the operator execution through the RPC protocol. The client part includes RPC call, operator execution log and execution result query function. RPC call is to call the operator on the server. The operator execution log records various logs generated by the operator call. The execution result query is to query the result of this operator execution. The server operator execution includes operator loading, memory management, and execution result saving functions. Then the operator is loaded and the operator information is loaded into the memory. The memory management monitors the memory and performs Release or save operation, save the execution result of the server and save it to the data access plug-in. The stand-alone execution includes two parts: the client and the server. The client includes RPC call, operator information query, execution result query, and operator execution log function. The RPC call is to call the server to execute the operator function, and the operator information is queried in real time. The execution result query is to query the execution result of the operator this time. The operator execution log records the log generated by the operator call. The server-side operator execution imports the corresponding execution entry module of the operator through the class loading function, calls the entry module, and passes in parameters to execute the operator, and returns the execution result to the client.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A parallel computing task process planning platform, characterized in that: It includes a task planning process design unit and a parallel computing task management unit, wherein the task planning process unit includes a dynamic operator, a task triggering module, a task instance management module and a task process management module; The task planning process unit is started by designing the process. The user edits the task process on the front-end page. The task process is represented by a node representing the algorithm operator. Each node contains execution parameters. Multiple nodes are displayed through a directed acyclic graph, and the execution order is represented by arrow relationships. The task process management module monitors the task process start event, executes the dynamic operator in the order defined by the process, and changes the state of the node in the task flow chart DAG according to the execution result of the dynamic operator. When the algorithm is successfully executed, the subsequent node state is changed to execution success. When the algorithm fails to execute, the subsequent node status is changed to execution failure; the task instance management module includes task instance creation, task instance information query and task instance deletion functions.

2. A parallel computing task process planning platform as claimed in claim 1, characterized in that: The template submodule built into the dynamic operator includes two functions: operator management and task flow chart management based on operator combination sorting, wherein operator management includes querying built-in operators, querying custom operators, creating custom operators, modifying custom operators, and deleting custom operators; Built-in operators are operators built into the system, and custom operators are operators created by users. Built-in operators correspond to static operators in the system and are automatically loaded into the system when the system starts; custom operators correspond to dynamic operators in the system and are created, modified, and deleted by users and updated to the system in real time.

3. A parallel computing task process planning platform as claimed in claim 1, characterized in that: When the task trigger module receives a request, the user enters the parameters required for task triggering in the front-end interface, creates a task instance in the queue state, and then executes the algorithm. In the new process allocated by the task scheduling process, based on the operator name passed by the task flow management submodule, the algorithm execution entry function module path associated with the operator is obtained, and through parallel computing task management, the algorithm execution entry function module path and request parameters are passed in for execution.

4. A parallel computing task process planning platform as claimed in claim 1, characterized in that: The parallel computing task management unit includes four functions: operator encapsulation, operator execution, node and plug-in management. The operator encapsulation encapsulates the user-written code into an executable operator and installs it in a distributed environment, and simultaneously supports RPC remote calls; the operator encapsulation includes two parts: code template and distributed environment configuration, wherein the code template part provides editable code, operator information editing, request parameter configuration, and RPC parameter configuration functions; wherein the distributed environment configuration is to install the third-party Python library required for the operator in a distributed environment, and install the relevant library through plug-in installation. After the installation is completed, the operating environment of the operator is met.

5. A parallel computing task process planning platform as claimed in claim 4, characterized in that: The operator execution part includes two parts: distributed execution and single-machine execution. The distributed execution part includes two parts: client and server. The client calls the server to execute the operator and returns the result of the operator execution through the RPC protocol. The client part includes RPC call, operator execution log and execution result query function. RPC call is to call the operator on the server. The operator execution log records various logs generated by the operator call. The execution result query is to query the result of this operator execution. The server operator execution includes operator loading, memory management, and execution result saving functions. Then the operator is loaded and the operator information is loaded into the memory. The memory management monitors the memory and performs Release or save operation, save the execution result of the server and save it to the data access plug-in. The stand-alone execution includes two parts: the client and the server. The client includes RPC call, operator information query, execution result query, and operator execution log function. The RPC call is to call the server to execute the operator function, and the operator information is queried in real time. The execution result query is to query the execution result of the operator this time. The operator execution log records the log generated by the operator call. The server-side operator execution imports the corresponding execution entry module of the operator through the class loading function, calls the entry module, and passes in parameters to execute the operator, and returns the execution result to the client.

Images