Workflow execution method and system based on distributed architecture

Abstract

The application discloses a kind of based on distributed architecture's workflow execution method and system, the method includes the following steps: S1. when workflow node execution request, load node strategy configuration, and check whether mandatory rule belongs to mandatory independent execution or mandatory embedded execution, if yes, enter step S3, otherwise enter step S2;S2. initialization decision score, layer by layer calculation and add / subtract score, to form final decision score and determine independent execution mode or embedded execution mode;S3. after determining independent execution mode or embedded execution mode, according to the mode path of correspondence, carry out workflow.This application introduces "embedded execution mode" and "independent execution mode" for the execution of workflow node, dynamically decides its execution mode, and simultaneously, by introducing the mode of strategy configuration file, realizes resource restriction and resource permission constraint under "embedded execution mode" and "independent execution mode".

Description

Technical Field

[0001] This invention relates to the field of workflow software system technology, and in particular to a workflow execution method and system based on a distributed architecture. Background Technology

[0002] Workflow systems are core infrastructure for modern enterprises to automate business processes and process data. By breaking down complex business logic into a series of interdependent, executable nodes (tasks) and scheduling them according to predefined rules, they greatly improve productivity and system reliability.

[0003] However, as business complexity and scale increase, traditional workflow systems are gradually revealing their limitations when facing real-world business challenges:

[0004] 1. Limited Execution Modes and Lack of Flexibility and Adaptability: Traditional workflow engines, such as Apache Airflow, Zeebe, and Temporal, either embed all nodes within the main process or use independent processes or containers for execution. While embedded execution offers advantages like low latency and high efficiency, it suffers from insufficient resource isolation. When a node consumes significant memory or CPU resources, it can easily impact the stability of the entire workflow engine. Independent execution, while providing good resource isolation, introduces additional network communication overhead and resource scheduling complexity, making it overly cumbersome for lightweight nodes.

[0005] 2. Insufficient Distributed Capabilities and Coarse-grained Resource Control: To address single-point performance issues, some workflow systems have evolved into distributed architectures, deploying the engine across multiple server nodes. However, this distributed approach typically remains at the "engine level" for load balancing, with node execution still strongly tied to the engine process. System administrators can limit resources for the entire engine process, but they cannot implement fine-grained resource quotas for the hundreds or thousands of concurrently executing nodes from different business processes within the engine (e.g., limiting the "transcoding node" of business A to a maximum of 0.5 CPU cores and 256MB of memory). This leads to the "noisy neighbor" problem, where high-load business processes crowd out the resources of low-load processes, failing to achieve true multi-tenant resource isolation.

[0006] 3. Weak security isolation mechanisms, making it difficult to cope with complex execution environments: In modern enterprises, workflow nodes may be developed by different teams, and may even execute third-party or user-submitted code scripts. In traditional embedded execution models, all code shares the context of the engine process, meaning that malicious or vulnerable code on one node can easily access data on other nodes, and even perform high-risk operations such as file reading / writing and network attacks on the host machine. Existing solutions typically rely on coding standards and code reviews, lacking mandatory and reliable technical isolation measures, resulting in extremely high security risks. Although some systems attempt to use independent processes or containers to execute all tasks, this introduces new problems: for massive amounts of simple tasks, the overhead (time and resources) of creating containers far exceeds the execution overhead of the tasks themselves, leading to a significant decrease in system throughput and a significant increase in latency, making it counterproductive.

[0007] 4. Static scheduling strategies lack dynamic awareness of task characteristics and system status: Existing system scheduling strategies are often static or based on simple rules (such as round-robin or least connections). When deciding which worker node to assign a task to, they do not consider the resource requirements of the task itself (whether it is CPU-intensive or I / O-intensive) or the real-time load status of the current worker node (CPU, memory usage, task queue backlog). This "blind" scheduling leads to uneven system load, with some nodes overloaded while others are idle, resulting in low overall resource utilization. Furthermore, it fails to proactively implement degradation strategies (such as isolating high-load tasks to an isolated environment) to ensure the stability of core services when the system faces overload risks. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a workflow execution method and system based on a distributed architecture. It introduces "embedded execution mode" and "independent execution mode" for the execution of workflow nodes, dynamically decides their execution mode, and implements resource restrictions and resource permission constraints under "embedded execution mode" and "independent execution mode" by introducing a policy configuration file.

[0009] The objective of this invention is achieved through the following technical solution: a workflow execution method based on a distributed architecture, comprising the following steps:

[0010] S1. When a workflow node executes a request, it loads the node policy configuration and checks whether the enforcement rule belongs to forced independent execution or forced embedded execution:

[0011] If forced to execute independently, select the independent execution mode and proceed to step S3;

[0012] If embedded execution is forced, select the embedded execution model and proceed to step S3;

[0013] If it is neither mandatory independent execution nor mandatory embedded execution, then proceed to step S2;

[0014] S2. Initialize the decision score, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode;

[0015] S3. After determining the standalone execution mode or the embedded execution mode, execute the workflow according to the corresponding mode path.

[0016] A workflow execution system based on a distributed architecture includes:

[0017] The strategy configuration and enforcement rule determination unit is used to load the node strategy configuration and check whether the enforcement rule belongs to forced independent execution or forced embedded execution when the workflow node executes the request.

[0018] The decision score superposition and execution mode determination unit is used to initialize the decision score when the mandatory rule does not belong to mandatory independent execution or mandatory embedded execution, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode.

[0019] The workflow execution unit is used to execute the workflow according to the corresponding mode path after determining the independent execution mode or the embedded execution mode.

[0020] The beneficial effects of this invention are: this invention introduces "embedded execution mode" and "independent execution mode" for the execution of workflow nodes. During runtime, the execution mode is dynamically determined according to the algorithm flow described below. At the same time, by introducing a strategy configuration file, resource restrictions and resource permission constraints are implemented under "embedded execution mode" and "independent execution mode". Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0023] like Figure 1 As shown, a workflow execution method based on a distributed architecture includes the following steps:

[0024] S1. When a workflow node executes a request, it loads the node policy configuration and checks whether the enforcement rule belongs to forced independent execution or forced embedded execution:

[0025] If forced to execute independently, select the independent execution mode and proceed to step S3;

[0026] If embedded execution is forced, select the embedded execution model and proceed to step S3;

[0027] If it is neither mandatory independent execution nor mandatory embedded execution, then proceed to step S2;

[0028] After the workflow instance is initialized, the system begins executing its nodes according to the workflow's flow design. Whenever a specific workflow node is reached, the system first performs the "Load Node Policy Configuration" step (if it hasn't been loaded yet): reading the policy configuration file corresponding to the node X being executed. This policy configuration file declares the node's resource constraints and permission permissions in a structured configuration manner under two execution modes (embedded execution and remote execution).

[0029] The resource constraints of the embedded execution module include: the configured memory limit, the maximum CPU execution time Tcpu_max, and the maximum wall execution time Twall_max;

[0030] Resource constraints for the independent execution model include: configured memory limit, maximum number of shared CPU cores, and maximum number of dedicated CPU cores;

[0031] Permissions in both embedded and standalone execution models include: allowed directories of files to be read, allowed directories of files to be written, allowed domain names or IP addresses to be accessed, and allowed environment variables to be accessed.

[0032] For example, in embedded execution mode, node X's resource quota is a maximum heap memory (V8) limit of 256 MB, a young generation limit of 64 MB, a maximum CPU execution time (Tcpu_max) of 10 seconds, and a total wall time limit (Twall_max) of 30 seconds; in standalone execution mode, the container resource limits are a maximum memory limit of 512 MB, a CPU weight (cpuShares) of 512, and a maximum number of available CPU cores (cpus) of 0.5 (i.e., a maximum of 0.5 CPU cores can be used).

[0033] In addition to resource allocation, you can also configure resource permissions for node X, for example:

[0034] The set of allowed file paths for reading includes / data / images / , the set of allowed file paths for writing includes / data / thumbnails / , the set of allowed network domains for access includes test.domain.com, and the set of allowed environment variable keys includes TEST_ENV_NAME.

[0035] Each node can configure this policy file independently, which will then serve as the direct input for subsequent steps such as "pattern decision", "configuration synthesis", and "container startup / thread configuration".

[0036] After loading the strategy, the system proceeds to "Check Mandatory Rules" to determine whether the current node type belongs to the predefined "Mandatory Independent Execution List" (e.g., OCR recognition, big data computation, dynamic code execution, etc.). If the result is "yes" (belongs to mandatory independent execution), the independent execution mode is directly selected, and the process directly enters the independent execution path; the process ends at this branch, and no further scoring or other levels of evaluation are performed.

[0037] If the current workflow node type does not belong to the predefined "forced independent execution list", then determine whether it belongs to the predefined forced embedded execution list; if so, then directly select the embedded execution mode and enter the embedded execution running path; the process ends at this branch and no further scoring or other level evaluations are performed.

[0038] If not, proceed to step S2.

[0039] S2. Initialize the decision score, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode;

[0040] If the "Check Mandatory Rules" judgment result is "No", the system initializes the decision score: Score = 0 (hereinafter referred to as Score or Score_independent). Then, the scores are calculated and accumulated / deducted layer by layer according to the following four-level evaluation order to form the final decision score;

[0041] First layer: Resource requirement assessment (CPU intensity determination and scoring)

[0042] Prioritizing the sources of CPU intensity on a node: The system checks and determines the sources of CPU intensity on a node in a strict order (the priority must be strictly in the following order):

[0043] (a) Has the density been explicitly set in the node configuration during workflow design? If so, use that setting, which has the highest priority.

[0044] (b) Otherwise, check the node's default attribute (density declared by the node developer during development); if it exists, use that default attribute, which has the second highest priority.

[0045] (c) If neither of the above two items exists, a statistical determination will be made at runtime. The statistical source is the sampling of the cumulative CPU usage time of the Cgroup to which the node belongs by Linux Cgroups (v1 or v2).

[0046] Runtime statistics methods (enabled only when the first two items are not specified):

[0047] Before the node is executed, record the wall clock time Tstart_wall and read the cumulative CPU usage time of the Cgroup in which the node process belongs (read from cpuacct.usage for Cgroup v1, and from usage_usec in cpu.stat for Cgroup v2), and record it as CPUstart.

[0048] (a) Before node execution: Record the current time, denoted as Tstart_wall ， Read the cumulative CPU usage time of the Cgroup to which the process belongs on this node. Cgroup v1 reads `cpuacct.usage`, and Cgroup v2 reads `usage_usec` from `cpu.stat`. (This is denoted as...) CPUstart .

[0049] (b) After the node is executed: Record the current time, denoted as Tend_wall ， Read the cumulative CPU usage time of the Cgroup where the process of this node belongs again. Cgroup v1 reads cpuacct.usage, and Cgroup v2 reads usage_usec from cpu.stat. Record it as CPUend.

[0050] (c) Calculate the difference: The total execution time of the node is The total CPU time consumed by the node value execution is .

[0051] (d) Calculate CPU intensity: CPU intensity D is defined as the ratio of CPU time to total execution time. Considering that nodes may use multi-core parallel computing, the execution time on the wall needs to be normalized by multiplying the available CPU cores Ncores for that node. The calculation formula is as follows:

[0052] Dnormalized=Tcpu / (Twall×Ncores )

[0053] in:

[0054] Ncores is the number of CPU cores allocated to this Cgroup.

[0055] The value of `Dnormalized` is typically between [0, 1]. A value closer to 1 indicates that the CPU is essentially "fully loaded" during node operation, representing a typical computationally intensive node. A value closer to 0 indicates that the node spends most of its time waiting for I / O, sleeping, or being preempted by other processes, representing a typical I / O-intensive or idle node.

[0056] Density Level Determination and Scoring Rules:

[0057] By setting two thresholds, Thresholdhigh and Thresholdlow, nodes can be divided into three levels:

[0058] High density: If Dnormalized ≥ Thresholdhigh.

[0059] Medium density: If Thresholdlow ≤ Dnormalized <Thresholdhigh。

[0060] Low density: If Dnormalized <Thresholdlow。

[0061] In practice, Thresholdhigh is typically initially set to 0.75 and Thresholdlow to 0.25. This means:

[0062] A node is considered high-density if, on average, each CPU core allocated to it spends more than 75% of its time computing during a single workflow task cycle.

[0063] If it's between 25% and 75%, it's considered medium density.

[0064] If it is below 25%, it is considered low density.

[0065] If the density source is "set during workflow design" or "preset node", the system will also apply the above scoring rules according to the set level.

[0066]

[0067] After completing the density assessment and score accumulation, the second level of evaluation begins.

[0068] Second layer: System status assessment (system load score)

[0069] Indicator collection:

[0070] The system monitors the current workload status of the workflow engine in real time, including CPU utilization, memory utilization, task queue backlog, and P95 execution latency. Load assessment uses a weighted scoring mechanism, comprehensively considering the deviation of current values ​​from historical baselines for each indicator. When the system load exceeds a preset threshold, the decision algorithm tends to select an independent running mode to reduce the pressure on the main process.

[0071] The system monitors the overall load status of the workflow engine in real time and evaluates it using a quantitative system load scoring model. The score ranges from 0 to 1, with a higher score indicating a higher system load and closer to overload.

[0072] The load assessment model integrates the following four key metrics:

[0073] CPU utilization (Mcpu): The CPU utilization of the engine's main process and the server it resides on.

[0074] Memory usage (Mmem): The memory usage of the engine's main process and the server it resides on.

[0075] Task queue backlog (Mqueue): The ratio of the number of nodes to be executed to the number of worker threads.

[0076] P95 execution latency (Mlatency): The P95 execution time of the most recent N (e.g., 1000) completed embedded nodes.

[0077] Indicator normalization:

[0078] Since the metrics have different dimensions, they first need to be normalized to the [0, 1] interval, where 0 represents the ideal state and 1 represents the critical or overload state. A "warning threshold" (Twarn) and a "critical threshold" (Tcrit) are set for each metric Mi.

[0079]

[0080] For example, for CPU utilization, we can set Twarn_cpu=60% and Tcrit_cpu=85%. If the current CPU utilization is 70%, then its normalized score is (70-60) / (85-60)=0.4.

[0081] Weighted summation:

[0082] Assign a weight Wi to each normalized metric score Ni, representing the importance of that metric in assessing system load. The sum of all weights is 1 (∑Wi=1).

[0083] Where i can be cpu, mem, queue, or latency, then:

[0084] Ni represents Ncpu, Nmem, Nqueue, and Nlatency, which are the scores for CPU utilization, memory utilization, task queue backlog, and P95 execution latency, respectively.

[0085] Wi stands for Wcpu, Wmem, Wqueue, and Waitability, which represent the weights of CPU utilization, memory utilization, task queue backlog, and P95 execution latency, respectively.

[0086] The formula for calculating the total system load score Sload is as follows:

[0087] Sload=WcpuNcpu+WmemNmem+WqueueNqueue+WlatencyNlatency

[0088] The value of Sload ranges from [0,1], and the higher the value, the closer the system is to overload.

[0089] Convert the system load score into a decision score and accumulate them:

[0090] S core += Sload × 100

[0091] Therefore, when Sload = 0.7, it will contribute +70 points to the decision score. After completing the second layer, it will enter the third layer of evaluation.

[0092] Third layer: Network latency cost assessment (latency penalty)

[0093] For nodes that may choose to operate independently, the system attempts to estimate the end-to-end latency cost (Tnetwork) introduced by network communication. This cost is a key negative indicator for deciding whether to adopt the embedded mode.

[0094] The delay cost consists of the following components, calculated using the following formula:

[0095] Tnetwork=Thandshake+(Tserialize+Tdeserialize)+Ttransfer+Tremote_queue

[0096] in:

[0097] Connection establishment latency (Thandshake): The time required for the client to establish a TCP connection and perform a TLS handshake with the remote executor. Assuming a connection pool is available most of the time and connections can be obtained from it, this cost is approximately zero. If a new connection needs to be established, this cost equals the network round-trip time (RTT) overhead, obtained by periodically probing the target network path to obtain a dynamic average (this system measures this by maintaining a background ping).

[0098] Serialization / deserialization delay (Tserialize + Tdeserialize), where:

[0099] Tserialize represents the sum of the serialization time of the remote execution request and the serialization time of the remote execution result;

[0100] Tdeserialize represents the sum of the deserialization time of the remote execution request and the deserialization time of the remote execution result;

[0101] This latency is related to data size, data complexity, and the serialization framework. By actively sampling and statistically analyzing the data size of nodes at runtime, a linear regression model of data size -> serialization / deserialization time is established.

[0102] Data transfer latency (Ttransfer): The time required for data to be transmitted over the network. This is primarily determined by the amount of data and the available network bandwidth.

[0103]

[0104] EffectiveBandwidth is a variable set during system initialization based on the actual physical environment.

[0105] Remote queuing delay (Tremote_queue): The time a task waits in a queue after arriving at the remote executor due to the executor being busy. This is calculated by the node itself. Upon receiving a request, if the task enters the queue, the enqueue time is recorded. After dequeueing, the waiting time of the task in the queue is calculated and returned to the workflow engine along with the execution result.

[0106] Delay penalty rules (two levels of penalty):

[0107] If the estimated Tnetwork > 50 ms (a significant latency-aware threshold), then a penalty is imposed on the decision score. .

[0108] If the above conditions are met, and the data size is <1 MB and Tnetwork > 50 ms (i.e., small data size but high network latency), then an additional penalty will be applied. (At this point, the total penalties from the network are...) ).

[0109] After completing the third level, proceed to the fourth level of decision-making.

[0110] Fourth layer: Execution mode decision layer.

[0111] Decision threshold setting: Set the decision threshold Threshold_decision = 60.

[0112] Final comparison: If the final cumulative score S core If the threshold is greater than or equal to Thresholddecision, then the standalone execution mode is selected; otherwise, the embedded execution mode is selected. After selection, the corresponding execution path and resource / permission configuration are combined.

[0113] S3. After determining the standalone execution mode or the embedded execution mode, execute the workflow according to the corresponding mode path:

[0114] Execution after selecting the independent execution mode (containerized execution path)

[0115] If the system is forced into independent execution mode, or is determined to be in independent execution mode based on a scoring system, then the system enters the independent execution path. The specific process of this path is as follows:

[0116] Configuration and Container Parameter Translation:

[0117] Read the parameters of the independent execution mode resource constraints in the policy configuration file and translate them into container startup parameters;

[0118] Read the permissions settings in the policy configuration file:

[0119] Translate the allowed read and allowed write file directories into container mounting and network policies;

[0120] Translate allowed domain names or IPs into container-level network access whitelists / security groups or DNS / routing rules, and inject allowed access environment variables into container environment variables;

[0121] Select or generate a Seccomp configuration file to specify the whitelist for invocation based on permission requirements;

[0122] Container security and resource isolation:

[0123] Container runtime implements hard quotas (CPU, memory, I / O) through Linux kernel Cgroups (v1 / v2); achieves process / network / mount point isolation through namespaces (PID, Network, Mount, etc.); and loads custom system call whitelists through Seccomp to minimize the kernel attack surface. The container's root filesystem can be mounted in read-only mode according to a policy, and necessary data directories can be mounted in read-only mode or read-write mode as required by the policy.

[0124] Container startup command synthesis and scheduling:

[0125] The complete container startup command text is hashed (SHA-256), and this hash value, along with the label, is used as an index to search for an existing container node cluster with the same label. If a matching container node cluster exists, an execution request is sent to that cluster; otherwise, the corresponding configured container node cluster is initialized via the container runtime API, and an execution request is sent. Container scheduling, network access, and volume mounting are all handled in this step. After completion, containerized execution begins, and the container's running status is monitored until execution ends (process terminates).

[0126] Execution after selecting the embedded execution mode (Worker thread execution path)

[0127] If the system is forcibly identified as being in embedded execution mode, or is determined to be in embedded execution mode through scoring, then the system enters the embedded execution path. The specific implementation and control flow of this path are as follows:

[0128] The resource constraints of the embedded execution module are read from the strategy configuration file: the configured memory limit, Tcpu_max, Twall_max, etc., are used as runtime parameters and external monitoring thresholds when creating worker threads (using worker_threads in Node.js as an example). For example, when creating worker threads, the V8 layer limits the heap memory limit (paraphrasing the example values ​​as: setting the V8 heap memory limit for worker threads to maxOldGenerationSizeMb = 256 MB, maxYoungGenerationSizeMb = 64 MB), so that if memory attempts to exceed this limit, V8 throws an exception causing the thread to crash, thus achieving hard isolation of memory.

[0129] CPU and monitoring of execution freezes (software control and watchdog):

[0130] An event loop monitoring and heartbeat mechanism are used to detect CPU usage and system crashes. Specific implementation steps:

[0131] (1) Initialization: When the main thread creates a Worker, it retrieves Tcpu_max and Twall_max from the policy configuration and sets the two thresholds accordingly. The main thread starts a total wall duration timer for the Worker (e.g., via setTimeout) with a duration of Twall_max; if the task has not finished by the timer expires, the timeout circuit breaker is triggered and the execution of the Worker is terminated.

[0132] (2) Worker thread heartbeat: A high-priority heartbeat timer decoupled from business logic is started inside the Worker. The heartbeat periodically (e.g., every second or at intervals specified by the strategy) reads the CPU execution time statistics file of the current thread in the Linux kernel; calculates the CPU time increment of the current thread since the last heartbeat; and sends the heartbeat packet, including the cpuTime field (i.e., the cumulative CPU usage time T_cpu_current), to the main thread through the message channel.

[0133] (3) Main thread decision-making: The main thread resets a heartbeat timeout timer (e.g., 5 seconds) each time it receives a heartbeat packet; if the heartbeat timeout timer expires, it determines that the event loop may be blocked (a false death) and triggers the circuit breaker. After receiving a heartbeat packet, the main thread checks the cumulative CPU time T_cpu_current; if T_cpu_current > Tcpu_max, it triggers the CPU over-limit circuit breaker. Through this mechanism, memory overflow is guaranteed by the V8 layer, while CPU abuse and event loop death are guaranteed by an external monitoring mechanism.

[0134] Dynamic permission adaptation based on runtime characteristics:

[0135] In embedded mode, the system probes the current runtime type (Deno, Node.js, or Bun) before creating a Worker, and adopts different permission adaptation strategies based on the probe results:

[0136] (1) If it is a Deno runtime, the policy permissions are precisely translated into Deno permission objects and passed in when creating a Worker (for example, the permission to read / data / images / is converted into a Deno permission object read:[" / data / images / "]). The Deno runtime forcibly locks the permissions at the sub-thread level, which cannot be bypassed by the code, thereby achieving the highest level of embedded mode security.

[0137] (2) For Node.js or Bundle, a "best-effort" proxy isolation method is adopted: a bootstrap script is injected when the Worker starts, the script initializes the permission manager and receives the permission policy assigned by the main thread; the critical functions of high-risk modules (such as node:fs, node:child_process) are overridden (using Object.defineProperty or Proxy) to check whether the operation parameters are in the allowed list before execution; if they are not in the allowed list, an "access denied" exception is thrown. This method can defend against most non-compliant calls, but there is still a risk of deliberate bypass through underlying process.binding and other methods, hence the name "best-effort mode".

[0138] Worker Acquisition and Recycling:

[0139] The system retrieves a Worker that meets the policy requirements from the thread pool (or creates one if none exists), injects the aforementioned resource limits, heartbeat, and permission policies, and starts execution until the task is completed or the monitoring mechanism triggers a circuit breaker. After execution, the Worker is recycled and execution statistics (including actual Tcpu, Twall, serialized data size, network interaction statistics, etc.) are reported for subsequent runtime statistics and model training.

[0140] A workflow execution system based on a distributed architecture includes:

[0141] The strategy configuration and enforcement rule determination unit is used to load the node strategy configuration and check whether the enforcement rule belongs to forced independent execution or forced embedded execution when the workflow node executes the request.

[0142] The decision score superposition and execution mode determination unit is used to initialize the decision score when the mandatory rule does not belong to mandatory independent execution or mandatory embedded execution, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode.

[0143] The workflow execution unit is used to execute the workflow according to the corresponding mode path after determining the independent execution mode or the embedded execution mode.

[0144] The above description represents preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technical or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A workflow execution method based on a distributed architecture, characterized in that: Includes the following steps: S1. When a workflow node executes a request, it loads the node policy configuration and checks whether the enforcement rule belongs to forced independent execution or forced embedded execution: If forced to execute independently, select the independent execution mode and proceed to step S3; If embedded execution is forced, select the embedded execution model and proceed to step S3; If it is neither mandatory independent execution nor mandatory embedded execution, then proceed to step S2; S2. Initialize the decision score, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode; S2 includes: S201. Initialize decision score: Score = 0; S202. First-level evaluation: Determine the priority of the source of node CPU density and accumulate decision scores; CPU density is defined as the ratio of CPU time to total execution time, and is divided into high density, medium density and low density; S203. Second-level evaluation: Real-time monitoring of the current workflow engine's load status, determining the load score and converting it into a decision score for accumulation; S204. Third-level evaluation: Estimate the end-to-end delay cost introduced by network communication, and accumulate the decision score penalty to obtain the final decision score; the penalty accumulation refers to subtracting the penalty score from the decision score; S205. Fourth-level decision: Compare the final decision score with the decision threshold Threshold_decision. If the final cumulative score Score ≥ Threshold_decision, select the independent execution mode; otherwise, select the embedded execution mode. S3. After determining the standalone execution mode or the embedded execution mode, execute the workflow according to the corresponding mode path.

2. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: S1 includes: S101. When a workflow node executes a request, it loads the node policy configuration, including the policy configuration file corresponding to the workflow node. The policy configuration file declares the resource constraints and permission permissions of the node in two execution modes in a structured configuration manner. The two execution modes include embedded execution mode and independent execution mode. The resource constraints of the embedded execution module include: the configured memory limit, the maximum CPU execution time Tcpu_max, and the maximum wall execution time Twall_max; Resource constraints for the independent execution model include: configured memory limit, maximum number of shared CPU cores, and maximum number of dedicated CPU cores; Permissions in both embedded and standalone execution models include: allowed directories of files to be read, allowed directories of files to be written, allowed domain names or IP addresses to be accessed, and allowed environment variables to be accessed. S102. Determine whether the current workflow node type belongs to the predefined list of mandatory independent executions: If so, then directly select the independent execution mode and enter the independent execution path; If not, proceed to step S103; S103. Determine whether the current workflow node type belongs to the predefined forced embedded execution list; If so, directly select the embedded execution mode and enter the embedded execution running path; If not, proceed to step S2.

3. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: S202 includes: A1. Determine the priority of the source of node CPU density: A11. When designing the workflow, has the density been explicitly set in the node configuration, including high density, medium density, and low density? If it exists, use that setting; otherwise, proceed to step A12. A12. Check the node's preset attributes, including high density, medium density, and low density; if they exist, use that preset, which has the second highest priority. A13. If neither of the above two items exists, a statistical determination will be performed at runtime. The runtime statistical methods include: (a) Before node execution: Record the current time, denoted as Tstart_wall ， Read the cumulative CPU usage time of the Cgroup where the process of this node is located, and record it as CPUstart; (b) After the node is executed: Record the current time, denoted as Tend_wall ， Read the cumulative CPU usage time of the Cgroup where the process of this node is located again, and record it as CPUend; (c) Calculate the difference: The total execution time of the node is The total CPU time consumed by the node value execution is ； (d) Calculate CPU intensity: CPU intensity D is defined as the ratio of CPU time to total execution time. Considering that the node uses multi-core parallel computing, the execution time needs to be normalized by multiplying the time on the wall by the number of CPU cores available to the node, Ncores. The calculation formula is as follows: Dnormalized=Tcpu / (Twall×Ncores ) in: Ncores is the number of CPU cores allocated to this Cgroup; A2. Denseness Level Determination and Scoring Rules: By setting two thresholds, Thresholdhigh and Thresholdlow, nodes are divided into three levels: High density: If Dnormalized ≥ Thresholdhigh; Medium density: If Thresholdlow ≤ Dnormalized <Thresholdhigh; Low density: If Dnormalized <Thresholdlow; A3. Accumulate decision scores: ； If the density source is "set during workflow design" or "preset node", the system will also apply the above scoring rules according to the set level.

4. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: S203 includes: B1. Indicator Collection: Monitor the current workload status of the workflow engine in real time, including CPU utilization, memory utilization, task queue backlog, and P95 execution latency. B2. Since the dimensions of each indicator are different, they first need to be normalized to the interval [0, 1], where 0 represents the ideal state and 1 represents the critical or overload state; set a warning threshold Twarn and a critical threshold Tcrit for each indicator Mi. ； B3. Weighted summation: Assign a weight Wi to each normalized metric score Ni, representing the importance of that metric in evaluating system load, with the sum of all weights being 1; Where i can be cpu, mem, queue, or latency, then: Ni can be Ncpu, Nmem, Nqueue, or Nlatency, representing the scores of CPU utilization, memory utilization, task queue backlog, and P95 execution latency, respectively. Wi stands for Wcpu, Wmem, Wqueue, or Flatity, which represent the weights of CPU utilization, memory utilization, task queue backlog, and P95 execution latency, respectively. Total Load Score (Sload) Calculation Formula: Sload=WcpuNcpu+WmemNmem+WqueueNqueue+WlatencyNlatency The value of Sload ranges from [0,1], and a higher value indicates that the system is closer to overload; Convert the system load score into a decision score and accumulate them: 。 5. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: S204 includes: The estimated end-to-end delay cost introduced by the node due to network communication consists of the following components, calculated using the following formula: Tnetwork=Thandshake+(Tserialize+Tdeserialize)+Ttransfer+Tremote_queue Thandshake is the time required for the client to establish a TCP connection and perform a TLS handshake with the remote executor. Tserialize + Tdeserialize is the serialization / deserialization delay, where: Tserialize represents the sum of the serialization time of the remote execution request and the serialization time of the remote execution result; Tdeserialize represents the sum of the deserialization time of the remote execution request and the deserialization time of the remote execution result; Ttransfer is the time required for data to be transmitted over the network; Tremote_queue is the time a task waits in the queue after arriving at the remote executor because the executor is busy. Delay penalty rules: (1) If the estimated Tnetwork > 50 ms, then a penalty is applied to the decision score. ; (2) If (1) holds true, and the data size is < 1 MB and Tnetwork > 50 ms, then an additional penalty will be applied. .

6. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: In step S3, if the system is forcibly determined to be in independent execution mode, or is determined to be in independent execution mode by scoring, then the system enters the independent execution path, including: C1. Configuration and Container Parameter Translation: Read the parameters of the independent execution mode resource constraints in the policy configuration file and translate them into container startup parameters; Read the permissions settings in the policy configuration file: Translate the allowed read and allowed write file directories into container mounting and network policies; Translate allowed domain names or IPs into container-level network access whitelists / security groups or DNS / routing rules, and inject allowed access environment variables into container environment variables; Select or generate a Seccomp configuration file to specify the whitelist for invocation based on permission requirements; C2. Container Security and Resource Isolation: Containers implement hard quotas through Linux kernel Cgroups during runtime; Isolation of processes, networks, and mount points is achieved through namespaces; Seccomp loads a custom system call whitelist to minimize the kernel attack surface; the container's root file system is mounted in read-only mode according to the policy, and necessary data directories are mounted in read-only mode or read-write mode as required by the policy. C3. Container startup command synthesis and scheduling: Calculate the hash value of the complete container startup command text, and use the hash value and label as an index to find container node clusters with the same label in the cluster. If a matching cluster of container nodes exists, an execution request is sent to that cluster. If it does not exist, the corresponding configured container node cluster will be initialized through the container runtime API and a request will be sent for execution. This step handles container scheduling, network access, and volume mounting. Once completed, it proceeds to containerized execution and monitors the container's running status until execution ends.

7. The workflow execution method based on a distributed architecture according to claim 1, characterized in that: In step S3, if the system is forcibly determined to be in embedded execution mode, or is determined to be in embedded execution mode by scoring, then the embedded execution path is entered, including: D1. Read the resource constraints of the embedded execution module in the strategy configuration file, and use the configured memory limit, Tcpu_max and Twall_max as the running parameters and external monitoring thresholds when creating Worker threads; D2, CPU and execution freeze monitoring: CPU usage and apparent crash detection are implemented using an event loop monitoring and heartbeat mechanism. Specific implementation steps are as follows: D21. Initialization: When the main thread creates a Worker, it retrieves Tcpu_max and Twall_max from the policy configuration and sets two thresholds accordingly. The main thread starts a total wall duration timer for the Worker with a duration of Twall_max. If the task has not finished by the timer expires, the timeout circuit breaker is triggered and the execution of the Worker is terminated. D22. Worker Thread Heartbeat: A high-priority heartbeat timer, decoupled from business logic, is started inside the Worker. This heartbeat periodically reads the CPU execution time statistics file of the current thread in the Linux kernel, calculates the CPU time increment of the current thread since the last heartbeat, and sends the heartbeat packet, including the cpuTime field, to the main thread through the message channel. D23. Main Thread Decision: The main thread resets a heartbeat timeout timer each time it receives a heartbeat packet; if the heartbeat timeout timer expires, it determines that the event loop is blocked and triggers the circuit breaker; after receiving a heartbeat packet, the main thread checks the cumulative CPU time T_cpu_current in it; if T_cpu_current > Tcpu_max, it triggers the CPU over-limit circuit breaker. D3. Dynamic permission adaptation based on runtime characteristics: In embedded mode, the system probes the current runtime type before creating a Worker and adopts different permission adaptation strategies based on the probe results: D31. If it is a Deno runtime, the policy permissions are precisely translated into Deno permission objects and passed in when the Worker is created. The Deno runtime forcibly locks them at the sub-thread level, which cannot be bypassed by code, thus achieving the highest level of embedded mode security. D32. If it is Node.js or Bun, then adopt the "best-effort" proxy isolation: inject the bootstrap script when the worker starts, the script initializes the permission manager and receives the permission policy assigned by the main thread; check whether the operation parameters are in the allowed list before execution by overriding the key functions of high-risk modules; if they are not in the allowed list, throw an "no access permission" exception. D4. Worker Acquisition and Recycling: Obtain a Worker that meets the policy requirements from the thread pool. If none is found, create a new one. Inject the resource constraints, heartbeat, and permission policies mentioned above and start execution until the task is completed or the monitoring mechanism triggers a circuit breaker. After execution, reclaim the Worker and report the execution statistics for subsequent runtime statistics and model training.

8. A workflow execution system based on a distributed architecture, employing the method described in any one of claims 1 to 7, characterized in that: include: The strategy configuration and enforcement rule determination unit is used to load the node strategy configuration and check whether the enforcement rule belongs to forced independent execution or forced embedded execution when the workflow node executes the request. The decision score superposition and execution mode determination unit is used to initialize the decision score when the mandatory rule does not belong to mandatory independent execution or mandatory embedded execution, calculate and accumulate / deduct the score layer by layer to form the final decision score and determine the independent execution mode or the embedded execution mode. The workflow execution unit is used to execute the workflow according to the corresponding mode path after determining the independent execution mode or the embedded execution mode.

Images