A workflow application management method and system for containerized environment

Abstract

The application provides a kind of workflow application management method and system for containerization environment, it is related to computer technology field.The method includes the following steps, the structured front-end object model is converted to by visual workflow model construction module with the drag operation of user;Front-end object model is serialized as JSON format file by static logic inspection;JSON format file is sent into template engine by templating code generation module;JSON format file is automatically converted into executable workflow drive file by template engine.The application solves the prior art workflow system usually adopts static script or fixed template to arrange task, realizes the automatic conversion from workflow design to execution code generation by visual workflow model construction module and templating code generation module, reduces the technical threshold of complex workflow construction, improves system development efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a workflow application management method and system for containerized environments. Background Technology

[0002] With the development of scientific computing, data analysis, and complex engineering applications, workflow-based application organization has become an important technical means for managing complex computing tasks. Workflows divide complex computing tasks into multiple dependent node tasks and construct task execution flows through directed dependencies, thereby achieving automated execution and management of complex applications. In scenarios such as high-performance computing (HPC), scientific data processing, and artificial intelligence training, workflow technology is widely used in the organization and scheduling of multi-stage data processing, model training, and simulation computing tasks.

[0003] In recent years, containerization technologies (such as Docker and Kubernetes) have been increasingly applied to the deployment of scientific computing and complex application systems due to their excellent environment isolation, portability, and rapid deployment capabilities. By encapsulating applications and their runtime environments into container images, problems such as complex software dependencies, difficulty in reproducing environments, and challenges in cross-platform migration that exist in traditional application deployment processes can be effectively solved.

[0004] However, in the existing technology, existing workflow systems usually use static scripts or fixed templates for task orchestration, which lacks intuitive visual orchestration capabilities, resulting in low efficiency in building complex applications. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings of the prior art and provide a workflow application management method and system for containerized environments, solving the problems caused by the deep integration of containerization technology and workflow management systems, such as lack of visual orchestration capabilities, difficulty in data exchange, and difficulty in deploying complex applications.

[0006] To achieve the above objectives, this invention proposes a workflow application management method and system for containerized environments.

[0007] In a first aspect, the present invention provides an application-driven job scheduling method based on a supercomputing internet, the method comprising: S101: The user's drag-and-drop operations are transformed into a structured front-end object model through the visual workflow model building module; the front-end object model is serialized into a JSON format file through static logic verification of the visual workflow model building module. S102: The JSON format file is sent to the template engine through the templated code generation module; the JSON format file is automatically converted into an executable workflow driver file through the template engine; a bidirectional traceability relationship is established between the generated code of the JSON format file and the workflow driver file; S103: Deploy workflow driver files to a container environment using application deployment methods; S104: Data interaction is performed during the execution of workflow-driven files through data sharing methods; Among them, the process of executing steps S101-S104 includes S201: S201: Achieve stable operation and self-healing of workflow applications through full-process monitoring and automated lifecycle management.

[0008] A further improvement of the present invention is that the visualization workflow model construction module includes a front-end object model, static logic verification, port compatibility verification, and mandatory parameter integrity check. The front-end object model is organized using a graph data structure, and can be formally defined as a triple; the front-end object model can be represented as M=(N,L,C), where: M stands for Front-end Object Model; N={n1,n2,...n m Let} be a set of nodes, where each node is n. i =(id i ,type i ,params i ,pos i This includes the node's identifier, node type, node parameter set, and node position coordinates; L={l1,l2,...l k} represents the set of connections, where each connection l j =(src j ,dst j branch j This indicates a data or control dependency from the source node to the target node; C represents the canvas metadata, which includes information such as process name, process description, and global process variables. The static logic verification uses a topology sorting algorithm to detect whether the directed graph corresponding to the front-end object model is a directed acyclic graph; the port compatibility verification ensures that the data types at both ends of each connection match; and the required parameter integrity check ensures that all necessary parameters have been configured.

[0009] A further improvement of this invention is that the templated code generation module includes a template engine, a code generation function, and automated code optimization and checking; the template engine is used to convert JSON format files into executable workflow-driven files; the code generation function is defined as G(J,T)=T T (J), where: J represents a JSON file, and T represents the type of the target execution template engine. T This refers to template transformation functions for template engine T; The automated code optimization and inspection ensure that the generated code is directly executable.

[0010] A further improvement of the present invention is that the application deployment method includes a dynamic volume mount deployment module; the dynamic volume mount deployment module includes an application identifier, a persistent storage sub-path, a VolumeMount mount list, and a container internal directory; the application identifier calculates the mapping relationship of the persistent storage sub-path; the VolumeMount mount list dynamically maps the persistent storage sub-path to the container internal directory.

[0011] A further improvement of the present invention is that the data sharing method includes a scene-aware data interaction module; the scene-aware data interaction module includes the deployment locations of upstream and downstream nodes, the scale of data to be transmitted, the data access frequency, and a data interaction selection function; the deployment locations of upstream and downstream nodes are set to L; the scale of data to be transmitted is set to S; the data access frequency is set to P; the deployment locations of upstream and downstream nodes include those within the same node and those across nodes; The data interaction selection function includes cluster shared storage mode, node local shared memory mode, and cache middleware relay mode; The data interaction selection function is defined as SelectModel(L,S,F), where: If L = ∧S < θ1, then SelectModel(L,S,F) is a node-local shared memory mode; If L ≠ same node ∧S<θ2∧F>ϕ, then SelectModel(L,S,F) is the cache middleware transfer mode; Otherwise, SelectModel(L,S,F) is the cluster shared storage mode; Where θ1 is the first threshold of 1MB; θ2 is the second threshold of 100KB; and ϕ is the high-frequency access threshold.

[0012] A further improvement of the present invention is that the cluster shared storage mode is applicable to the large-scale data transmission scenario across nodes; the cluster shared storage mode includes a hierarchical directory structure, which is used to realize reliable data transmission in the cluster shared storage mode.

[0013] A further improvement of the present invention is that the node local shared memory mode is suitable for small data low-latency transmission scenarios within the same node; when the upstream and downstream nodes are deployed at the same node location, zero-copy data transmission is achieved through the node local shared memory mode.

[0014] A further improvement of the present invention is that the cache middleware relay mode is applicable to the cross-node high-frequency small data asynchronous communication scenario; the cache middleware relay mode includes cache middleware and data features; the cache middleware relay mode uses cache middleware such as Redis as a data relay hub; the cache middleware relay mode selects different data structures according to the data features.

[0015] Secondly, the present invention provides a workflow application management system for containerized environments, the system comprising a user interaction layer, a workflow management layer, an execution and resource management layer, and a data management layer; the user interaction layer, the workflow management layer, the execution and resource management layer, and the data management layer communicate and exchange data with each other through standardized interfaces; The user interaction layer includes a visual workflow model building module and a workflow configuration management module; the user interaction layer generates the JSON file through the visual workflow model building module. The workflow management layer includes a workflow parsing module, a code auto-generation module, and a container image encapsulation module; the workflow management layer generates executable workflow driver files through the templated code generation module. The execution and resource management layer includes a container scheduling and management module, a workflow execution engine module, and a runtime status monitoring module; the execution and resource management layer enables rapid deployment of workflow applications through the dynamic volume mounting and deployment module; The data management layer includes a data management module, a data interaction strategy module, and a data storage management module; the data management layer realizes data interaction of workflow applications during execution through the scene-aware data interaction module.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: (1) This invention realizes the automated conversion from workflow design to execution code generation through a visual workflow model building module and a templated code generation module, which reduces the technical threshold for building complex workflows and improves system development efficiency.

[0017] (2) This invention achieves the decoupling of application configuration and container image through the containerized deployment method of dynamic volume mounting deployment module, so that workflow application does not need to rebuild the image when the configuration changes, thereby improving the flexibility and efficiency of system deployment.

[0018] (3) The present invention can dynamically select the optimal data transmission mode according to the task running environment through the scene-aware data interaction module. In the same node computing environment, low-latency data exchange can be achieved through shared memory. In the cross-node environment, data transmission is completed through shared storage or cache middleware, thereby effectively improving the data interaction efficiency of the workflow system in cloud-native and high-performance computing environments. Attached Figure Description

[0019] Figure 1 This is a flowchart of a workflow application management method for containerized environments according to the present invention.

[0020] Figure 2 This is an architectural component diagram of a workflow application management system for containerized environments according to the present invention. Detailed Implementation

[0021] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0022] With the development of scientific computing, data analysis, and complex engineering applications, workflow-based application organization has become an important technical means for managing complex computing tasks. Workflows divide complex computing tasks into multiple dependent node tasks and construct task execution flows through directed dependencies, thereby achieving automated execution and management of complex applications. In scenarios such as high-performance computing (HPC), scientific data processing, and artificial intelligence training, workflow technology is widely used in the organization and scheduling of multi-stage data processing, model training, and simulation computing tasks.

[0023] In recent years, containerization technologies (such as Docker and Kubernetes) have been increasingly applied to the deployment of scientific computing and complex application systems due to their excellent environment isolation, portability, and rapid deployment capabilities. By encapsulating applications and their runtime environments into container images, problems such as complex software dependencies, difficulty in reproducing environments, and challenges in cross-platform migration that exist in traditional application deployment processes can be effectively solved.

[0024] However, deeply integrating containerization technology with workflow management systems still faces numerous challenges in the current technology landscape. On the one hand, existing workflow systems typically employ static scripts or fixed templates for task orchestration, lacking intuitive visual orchestration capabilities, resulting in low efficiency in building complex applications. On the other hand, in containerized execution environments, workflow tasks often require frequent data exchange and intermediate result transfer. Traditional data sharing methods (such as unified shared file systems or fixed interface communication) suffer from low efficiency, high coupling, and poor resource utilization in different computing environments. Furthermore, existing systems often require manual configuration of numerous runtime parameters and execution scripts during workflow application deployment, lacking automated application encapsulation and deployment mechanisms, making it difficult to meet the needs of rapid construction and flexible deployment of complex applications.

[0025] Therefore, there is an urgent need for a workflow application management method and system for containerized environments, which can realize the visual orchestration, automated code generation, containerized encapsulation and deployment, and flexible data interaction mechanism of workflow applications, thereby improving the construction efficiency, deployment flexibility and execution performance of complex applications.

[0026] To address the aforementioned technical issues, this invention proposes a workflow application management method and system for containerized environments. The core of this method lies in constructing a complete workflow application management mechanism, encompassing the entire process from workflow definition, code generation, containerized deployment to data interaction and operational monitoring. Example 1

[0027] like Figure 1 and Figure 2 As shown; Firstly, this embodiment provides a workflow application management method for containerized environments, the method comprising: S101: The user's drag-and-drop operations are transformed into a structured front-end object model through the visual workflow model building module; the front-end object model is serialized into a JSON format file through static logic verification of the visual workflow model building module. S102: The JSON format file is sent to the template engine through the templated code generation module; the JSON format file is automatically converted into an executable workflow driver file through the template engine; a bidirectional traceability relationship is established between the generated code of the JSON format file and the workflow driver file; S103: Deploy workflow driver files to a container environment using application deployment methods; S104: Data interaction is performed during the execution of workflow-driven files through data sharing methods; Among them, the process of executing steps S101-S104 includes S201: S201: Achieve stable operation and self-healing of workflow applications through full-process monitoring and automated lifecycle management.

[0028] In step S101, the visual workflow model building module includes a front-end object model, static logic verification, port compatibility verification, and mandatory parameter integrity check. The front-end object model is organized using a graph data structure, and it can be formally defined as a triple; the front-end object model can be represented as M=(N,L,C), where: M stands for Front-end Object Model; N={n1,n2,...n m Let} be a set of nodes, where each node is n. i =(id i ,type i ,params i ,pos i This includes the node's identifier, node type, node parameter set, and node position coordinates; L={l1,l2,...l k} represents the set of connections, where each connection l j =(src j ,dst j branch j This indicates a data or control dependency from the source node to the target node; C represents the canvas metadata, which includes information such as process name, process description, and global process variables. Static logic verification uses a topology sorting algorithm to check whether the directed graph corresponding to the front-end object model is a directed acyclic graph; port compatibility verification ensures that the data types at both ends of each connection match; and the completeness check of required parameters ensures that all necessary parameters have been configured.

[0029] The core of this visual workflow model building module is to convert the user's drag-and-drop operations on the canvas into a structured front-end object model in real time, and to perform static logic validation on the front-end object model to ensure its validity.

[0030] Specifically, the system monitors all user actions on the canvas in real time and maintains a front-end object model of a front-end flowchart in memory. This front-end object model is organized using a graph data structure and can be formally defined as triples: Definition 1: The front-end object model can be represented as M=(N,L,C), where: N={n1,n2,...n m Let} be a set of nodes, where each node is n. i =(id i ,typei ,params i ,pos i This includes the node's identifier, node type, node parameter set, and node position coordinates; L={l1,l2,...l k} represents the set of connections, where each connection l j =(src j ,dst j branch j This indicates a data or control dependency from the source node to the target node; C represents the canvas metadata, which includes information such as process name, process description, and global process variables.

[0031] After the front-end object model is built, the system automatically performs static logic verification. The core of the static logic verification is to use the topological sorting algorithm to detect whether the front-end object model has a directed acyclic graph.

[0032] The algorithm takes a workflow model M=(N,L) as input and outputs the validation results and error information. The specific steps are as follows: Calculate the in-degree of each node: indegree[n] = |{l∈L|l.dst=n}|; The initial queue Q contains all nodes where indegree[n] = 0; Initialize the visited node count to 0; While Q is not empty do Remove node n from Q; count = count + 1; For each connection l∈L originating from n, do Target node m = l.dst; indegree[m] = indegree[m] - 1; if indegree[m]=0 then Add m to Q; end if end for end while If count ≠ |N| then A circular dependency was detected, so a failure was returned and the circular path was marked. else Successful response.

[0033] The algorithm has a time complexity of O(N+L) and can efficiently detect circular dependencies in workflow definitions.

[0034] In addition to topology sorting, the system also performs port compatibility checks and mandatory parameter integrity checks to ensure that the data types at both ends of each connection match and that all necessary parameters are configured.

[0035] After successful verification, the system serializes the front-end object model into a standardized JSON format file, whose schema is defined as follows: This JSON file fully describes the workflow's node information, dependencies, and global configuration, providing standardized input for subsequent code generation.

[0036] In step S2, the templated code generation module includes a template engine, a code generation function, and automated code optimization and inspection. The template engine is used to convert JSON format files into executable workflow-driven files. The code generation function is defined as G(J,T)=T. T (J), where: J represents a JSON file, and T represents the type of the target execution template engine. T This refers to template transformation functions for template engine T; Automated code optimization and inspection ensure that the generated code is directly executable.

[0037] The core of this templated code generation module is to automatically convert standardized JSON format files into executable workflow-driven files through a template engine, while establishing a bidirectional traceability relationship between the generated code of the JSON format files and the workflow-driven files.

[0038] Specifically, the system pre-configures Jinja2 template libraries for different target execution engines (such as Apache Airflow), meaning it executes different template engines based on different targets. The code generation process can be formalized as follows: Definition 2: Let J be a JSON file and T be the template engine to be executed. Then the code generation function G(J,T) is defined as: G(J,T)=T T (J) Where T T This function is for template transformation of template engine T. It injects a JSON file into the template engine and generates the required target code.

[0039] During the template engine rendering process, the system iterates through the nodes array in the JSON format file and maps each node to the corresponding operator class according to its type.

[0040] This mapping relationship can be expressed as: Meanwhile, the system iterates through the `links` array, resolving the dependencies between nodes. For each link `l∈J.links`, it generates the corresponding dependency code: Dependency(l)=Operator(l.src)≫Operator(l.dst) This expression indicates that the target node task will be executed after the source node task has been completed.

[0041] To ensure the traceability of the generated code, the system calculates the SHA-256 hash value of the source JSON file and embeds it as a comment in the header of the generated code file: h=SHA-256(J) CodeHeader="# Source design version hash: "+h Finally, the system performs automated optimization and checks on the generated target code, as follows: First, the import statements are merged and standardized; second, a code formatting tool (such as Black) is used to perform PEP 8 style formatting; finally, basic syntax checks are performed to ensure that the generated code is directly executable.

[0042] In traditional containerized deployment practices, application configuration parameters are typically passed to the container in the following two ways: One approach is to package the configuration file directly into the image. In this method, the configuration file is written to the image layer via the COPY instruction in the Dockerfile. When the configuration needs to be modified (e.g., switching from a production environment to a testing environment), the following steps must be performed sequentially: modify the configuration file, rebuild the image, re-push the image, and update the container version. The image building and pushing process typically takes several minutes to tens of minutes, resulting in lengthy waits even for minor configuration changes.

[0043] Another approach is to pass the configuration via environment variables. While environment variables do not require rebuilding the image, modifying them still necessitates updating the Kubernetes Deployment configuration and triggering a container restart. A container restart means that running tasks are interrupted, which may lead to computational failures in batch processing scenarios and service unavailability in microservice scenarios.

[0044] The common problem with both methods is that no matter how minor the configuration change, the container must be restarted, and the former also incurs the additional overhead of image building and pushing. This problem becomes increasingly severe as the number of environments and applications increases, and the complexity of configuration management grows exponentially.

[0045] In step S3, the application deployment method includes a dynamic volume mount deployment module. This module includes an application identifier, persistent storage subpaths, a VolumeMount mount list, and container internal directories. The application identifier calculates the mapping relationship of the persistent storage subpaths. The VolumeMount mount list dynamically maps the persistent storage subpaths to the container internal directories. Specifically, the system calculates the mapping relationship of persistent storage sub-paths based on the application identifier, which can be formalized as follows: Definition 3: For application identifiers Its persistent storage subpath is defined as: in To represent different data types.

[0046] Based on the above path mapping, the system constructs a VolumeMount mount list, dynamically mapping persistent storage subpaths to directories inside the container. An example of its Kubernetes resource configuration is as follows: In this way, the application's configuration files, runtime data, and log files are all stored in a separate subpath of the persistent volume. When the configuration needs to be changed, only the configuration file in the corresponding persistent storage subpath needs to be updated; there is no need to rebuild the image, and the system will automatically load the latest configuration on the next startup.

[0047] Compared to traditional deployment methods, the improved deployment efficiency of this mechanism can be quantified as follows: In real-world production environments, image building and pushing typically take 2-5 minutes, while configuration synchronization can be completed in seconds, resulting in a significant efficiency improvement. It can be dozens of times greater.

[0048] The application configuration and container image are decoupled by using a dynamic volume mount deployment module. By using dynamic volume mount based on application identifiers, configuration changes can take effect without rebuilding the image, thereby greatly improving deployment efficiency.

[0049] In containerized execution environments, workflow tasks often require frequent data exchange and intermediate result transfer. Traditional data sharing methods (such as unified shared file systems or fixed interface communication) suffer from low efficiency, high coupling, and poor resource utilization in different computing environments. In step S4, the data sharing method includes a scene-aware data interaction module; the scene-aware data interaction module includes the deployment location of upstream and downstream nodes, the scale of data to be transmitted, the data access frequency, and the data interaction selection function; the deployment location of upstream and downstream nodes is set to L; the scale of data to be transmitted is set to S; the data access frequency is set to P; the deployment location of upstream and downstream nodes includes the same node and cross nodes; Data interaction selection functions include cluster shared storage mode, node local shared memory mode, and cache middleware relay mode; The data interaction selection function is defined as SelectModel(L,S,F), where: If L = ∧S < θ1, then SelectModel(L,S,F) is a node-local shared memory mode; If L ≠ same node ∧S<θ2∧F>ϕ, then SelectModel(L,S,F) is the cache middleware transfer mode; Otherwise, SelectModel(L,S,F) is the cluster shared storage mode; Where θ1 is the first threshold of 1MB; θ2 is the second threshold of 100KB; and ϕ is the high-frequency access threshold.

[0050] The core of the scene-aware data interaction mechanism lies in dynamically selecting the optimal data interaction mode based on the deployment location of task nodes, data scale, and access frequency, thereby achieving a balance between low latency and high throughput.

[0051] Cluster shared storage mode is suitable for large-scale data transmission scenarios across nodes; cluster shared storage mode includes a hierarchical directory structure, which is used to achieve reliable data transmission in cluster shared storage mode.

[0052] The node-local shared memory mode is suitable for low-latency data transmission scenarios within the same node; when upstream and downstream nodes are deployed in the same location, zero-copy data transmission can be achieved through the node-local shared memory mode.

[0053] The cache middleware relay mode is suitable for high-frequency, small-data asynchronous communication scenarios across nodes; the cache middleware relay mode includes cache middleware and data characteristics; the cache middleware relay mode uses cache middleware such as Redis as the data relay hub; the cache middleware relay mode selects different data structures according to data characteristics.

[0054] Specifically, the scene-aware data interaction module obtains the following information before the task node is executed: the deployment locations of upstream and downstream nodes. (Including same-node and cross-node data), the size of the data to be transmitted and data access frequency Based on this information, the system makes a decision according to preset rules, defining a data interaction mode selection function, which can be defined as: Definition 4: Data Interaction Mode Selection Function Defined as: SelectModel(L,S,F)= Where θ1 is the first threshold (usually 1MB) and θ2 is the second threshold (usually 100KB). This is the high-frequency access threshold.

[0055] The specific implementations of each mode are as follows: Mode A: Cluster Shared Storage Mode This model is suitable for large-scale data transmission scenarios across nodes; it achieves reliable data transmission by constructing a hierarchical directory structure. The data transmission delay in this mode can be modeled as follows: in For storage system bandwidth, The overhead of file system metadata operations.

[0056] Mode B: Node-local shared memory mode This mode is suitable for low-latency data transmission scenarios on the same node. When upstream and downstream components, i.e., upstream and downstream nodes, are deployed on the same node, zero-copy data transmission is achieved through shared memory. Upstream and downstream components create a shared memory segment through the shmget system call and attach it to the process address space through shmat; at the same time, POSIX semaphores are used to achieve read and write synchronization.

[0057] The transmission delay in this mode can be expressed as: T shared memory = T memory copies + T semaphores T shared memory = T memory copies + T semaphores in Typically in the microsecond range, In the nanosecond range. For data smaller than 1KB, shared memory is 2 to 3 orders of magnitude faster than shared storage.

[0058] Mode C: Cache middleware relay mode This model is suitable for high-frequency, small-data asynchronous communication scenarios across nodes. The system uses caching middleware such as Redis as a data transfer hub and selects different data structures based on data characteristics: The throughput of this mode can reach Operations per second, far exceeding file I / O operations on shared storage.

[0059] The core of this scenario-aware data interaction module lies in dynamically selecting the optimal data interaction mode based on the deployment location of upstream and downstream nodes of the task, the scale of data to be transmitted, and the frequency of data access, thereby achieving a balance between low latency and high throughput.

[0060] Finally, in step S201, stable operation and self-healing of workflow applications are achieved through full-process monitoring and automated lifecycle management.

[0061] Through the collaborative work of the four core modules mentioned above, this invention achieves automated management of workflow applications from definition and deployment to execution, effectively solving problems such as lack of visual orchestration capabilities, low deployment efficiency, limited data interaction, and imperfect lifecycle management in existing technologies.

[0062] In summary, the key technology of this invention lies in proposing an integrated management method for workflow applications in containerized environments. By unifying and integrating a visual workflow model building module, a templated code generation module, a dynamic volume mounting and deployment module, and a scene-aware data interaction module, the invention achieves automated management of workflow applications from definition and deployment to execution.

[0063] Specifically, this invention first constructs a visual workflow description based on a front-end process object model, and ensures the legality of the workflow structure through topology sorting and parameter validation mechanisms. Then, using a templated code generation module, it automatically converts the standardized process's JSON format file description into an executable workflow driver file, achieving automatic adaptation in multi-execution engine environments. In the application deployment phase, a dynamic volume mounting deployment module decouples the application configuration file from the container image. During the task execution phase, the system dynamically selects a data interaction mode based on the deployment location of upstream and downstream nodes, the size of the data to be transmitted, and the data access frequency, thereby constructing a multi-mode collaborative data interaction system. Through these mechanisms, this invention achieves automated management and operational optimization during the workflow application construction, definition, deployment, and execution processes.

[0064] Compared with existing technologies, this invention has significant advantages in workflow application management. First, through a visual workflow model building module and a templated code generation module, it achieves automated transformation from workflow design to execution code generation, reducing the technical threshold for building complex workflows and improving system development efficiency. Second, through the containerized deployment method of the dynamic volume mounting deployment module, it achieves decoupling between application configuration and container image, so that workflow applications do not need to rebuild the image when the configuration changes, thereby improving the flexibility and efficiency of system deployment. In addition, the scene-aware data interaction module proposed in this invention can dynamically select the optimal data transmission mode according to the task running environment. In the same node computing environment, it can achieve low-latency data exchange through cluster shared memory mode, and in the cross-node environment, it can complete data transmission through node local shared memory mode or cache middleware relay mode, thereby effectively improving the data interaction efficiency of the workflow system in cloud-native and high-performance computing environments.

[0065] Through the above-mentioned technical means, this invention realizes automated management of the entire lifecycle of workflow applications from definition and deployment to execution monitoring, and has good scalability and applicability.

[0066] Secondly, such as Figure 2 As shown in the figure, this embodiment 1 provides a workflow application management method system for containerized environments. The system includes a user interaction layer, a workflow management layer, an execution and resource management layer, and a data management layer. The user interaction layer, workflow management layer, execution and resource management layer, and data management layer communicate and exchange data with each other through standardized interfaces. The user interaction layer includes a visual workflow model building module and a workflow configuration management module; the user interaction layer generates JSON files through the visual workflow model building module. The workflow management layer includes a workflow parsing module, an automatic code generation module, and a container image packaging module; the workflow management layer generates executable workflow driver files through the templated code generation module. The execution and resource management layer includes a container scheduling and management module, a workflow execution engine module, and a runtime status monitoring module; the execution and resource management layer enables rapid deployment of workflow applications through a dynamic volume mounting and deployment module; The data management layer includes a data management module, a data interaction strategy module, and a data storage management module; the data management layer enables data interaction during the execution of workflow applications through the scenario-aware data interaction module.

[0067] Specifically, this invention first transforms user drag-and-drop operations into a structured front-end object model through a visual workflow model building module; then, through a templated code generation module, it automatically converts this front-end object model into an executable workflow-driven JSON file; subsequently, through a containerized deployment method using a dynamic volume mounting deployment module, it quickly deploys the JSON file to a container environment; during task execution, a scenario-aware data interaction module dynamically selects the optimal data interaction mode; and finally, through full-process monitoring and automated lifecycle management, it achieves stable operation and self-healing of the workflow application.

[0068] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A workflow application management method for containerized environments, characterized in that, The method includes: S101: The user's drag-and-drop operations are transformed into a structured front-end object model through the visual workflow model building module; the front-end object model is serialized into a JSON format file through static logic verification of the visual workflow model building module. S102: The JSON format file is sent to the template engine through the templated code generation module; the JSON format file is automatically converted into an executable workflow driver file through the template engine; a bidirectional traceability relationship is established between the generated code of the JSON format file and the workflow driver file; S103: Deploy workflow driver files to a container environment using application deployment methods; S104: Data interaction is performed during the execution of workflow-driven files through data sharing methods; Among them, the process of executing steps S101-S104 includes S201: S201: Achieve stable operation and self-healing of workflow applications through full-process monitoring and automated lifecycle management.

2. The workflow application management method for containerized environments according to claim 1, characterized in that, The visualization workflow model construction module includes a front-end object model, static logic verification, port compatibility verification, and mandatory parameter integrity check. The front-end object model is organized using a graph data structure, and can be formally defined as a triple. The front-end object model can be represented as M=(N,L,C), where: M stands for Front-end Object Model; N={n1,n2,...n m Let} be a set of nodes, where each node is n. i =(id i ,type i ,params i ,pos i This includes the node's identifier, node type, node parameter set, and node position coordinates; L={l1,l2,...l k } represents the set of connections, where each connection l j =(src j ,dst j branch j This indicates a data or control dependency from the source node to the target node; C represents the canvas metadata, which includes information such as process name, process description, and global process variables. The static logic verification uses a topology sorting algorithm to detect whether the directed graph corresponding to the front-end object model is a directed acyclic graph; the port compatibility verification ensures that the data types at both ends of each connection match; and the required parameter integrity check ensures that all necessary parameters have been configured.

3. The workflow application management method for containerized environments according to claim 2, characterized in that, The templated code generation module includes a template engine, a code generation function, and automated code optimization and checking; the template engine is used to convert JSON format files into executable workflow driver files; the code generation function is defined as G(J,T)=T T (J), where: J represents a JSON file, and T represents the type of the target execution template engine. T This refers to template transformation functions for template engine T; The automated code optimization and inspection ensure that the generated code is directly executable.

4. The workflow application management method for containerized environments according to claim 3, characterized in that, The application deployment method includes a dynamic volume mount deployment module; the dynamic volume mount deployment module includes an application identifier, a persistent storage sub-path, a VolumeMount mount list, and a container internal directory; the application identifier calculates the mapping relationship of the persistent storage sub-path; the VolumeMount mount list dynamically maps the persistent storage sub-path to the container internal directory.

5. The workflow application management method for containerized environments according to claim 4, characterized in that, The data sharing method includes a scene-aware data interaction module; the scene-aware data interaction module includes the deployment locations of upstream and downstream nodes, the scale of data to be transmitted, the data access frequency, and a data interaction selection function; the deployment locations of upstream and downstream nodes are set to L; the scale of data to be transmitted is set to S; the data access frequency is set to P; the deployment locations of upstream and downstream nodes include those within the same node and those across nodes; The data interaction selection function includes cluster shared storage mode, node local shared memory mode, and cache middleware relay mode; The data interaction selection function is defined as SelectModel(L,S,F), where; If L = ∧S < θ1, then SelectModel(L,S,F) is a node-local shared memory mode; If L ≠ same node ∧S<θ2∧F>ϕ, then SelectModel(L,S,F) is the cache middleware transfer mode; Otherwise, SelectModel(L,S,F) is the cluster shared storage mode; Where θ1 is the first threshold of 1MB; θ2 is the second threshold of 100KB; and ϕ is the high-frequency access threshold.

6. The workflow application management method for containerized environments according to claim 5, characterized in that, The cluster shared storage mode is suitable for large-scale data transmission scenarios across nodes; the cluster shared storage mode includes a hierarchical directory structure, which is used to achieve reliable data transmission in the cluster shared storage mode.

7. The workflow application management method for containerized environments according to claim 6, characterized in that, The node-local shared memory mode is suitable for low-latency data transmission scenarios within the same node; when the upstream and downstream nodes are deployed at the same node location, zero-copy data transmission is achieved through the node-local shared memory mode.

8. The workflow application management method for containerized environments according to claim 7, characterized in that, The cache middleware relay mode is suitable for the high-frequency, small-data asynchronous communication scenario across nodes; the cache middleware relay mode includes cache middleware and data features; The cache middleware relay mode uses cache middleware such as Redis as a data relay hub; the cache middleware relay mode selects different data structures according to the data characteristics.

9. A workflow application management system for containerized environments, characterized in that, The system is used to implement the workflow application management method for containerized environments as described in any one of claims 1 to 8. The system includes a user interaction layer, a workflow management layer, an execution and resource management layer, and a data management layer. The user interaction layer, the workflow management layer, the execution and resource management layer, and the data management layer communicate and exchange data with each other through standardized interfaces. The user interaction layer includes a visual workflow model building module and a workflow configuration management module; the user interaction layer generates JSON files through the visual workflow model building module. The workflow management layer includes a workflow parsing module, a code auto-generation module, and a container image encapsulation module; the workflow management layer generates executable workflow driver files through the templated code generation module. The execution and resource management layer includes a container scheduling and management module, a workflow execution engine module, and a runtime status monitoring module; the execution and resource management layer enables rapid deployment of workflow applications through the dynamic volume mounting and deployment module; The data management layer includes a data management module, a data interaction strategy module, and a data storage management module; the data management layer realizes data interaction of workflow applications during execution through the scene-aware data interaction module.

Images