A design pattern-based DAG operator extension method, system and device

Abstract

The application relates to the technical field of computer software, and discloses a DAG operator extension method, system and device based on a design pattern, which comprises the following steps: based on each original operator class in a workflow, a decorator pattern is used to associate and register a business logic class with the original operator class to generate an operator class system; a factory pattern is used to determine the proxy instance creation mode corresponding to each original operator class, and a dependency injection pattern is used to scan, instantiate and container register the factory class to obtain operator factory instances corresponding to each operator; an agent pattern is used to control and enhance the operator execution process of the agent class; a chain of responsibility pattern is used to sequentially load the operator configuration information required by each original operator class; the corresponding operator factory instance is obtained from a dependency injection container, and a corresponding proxy instance is generated to complete the execution of all operators of the workflow, and an output result is returned. The application significantly improves the expansibility and flexibility of a workflow system.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and more specifically, to a method, system, and device for expanding DAG operators based on design patterns. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and cloud computing technologies, workflow orchestration and operator extension based on Directed Acyclic Graphs (DAGs) have become core components of modern software systems, especially data processing and AI application development platforms. In such systems, operators, as the basic functional units in the workflow, directly impact the overall system capability due to their ease of extension, functional enhanceability, and architectural maintainability. Therefore, ensuring system flexibility while achieving convenient operator extension, unified management, and reliable operation has become a key technical challenge for the continued development of DAG workflow platforms.

[0003] Existing DAG workflow platforms already offer a certain level of operator orchestration capabilities. For example, AI application development platforms like Dify typically use a visual drag-and-drop approach to build workflows and have built-in various model invocation and data processing operators. However, adding new operators often requires adherence to the platform's pre-defined interface specifications and implementation methods, making the expansion process highly dependent on the platform's mechanisms. Similarly, container-based workflow engines like Argo Workflow typically define workflow steps using YAML and execute each step as an independent container. While they possess strong task orchestration capabilities, when adding new operators, integration usually requires combining image building, deployment, and configuration processes, resulting in a relatively long development and maintenance chain. Overall, existing technologies focus more on workflow orchestration and execution.

[0004] Existing DAG workflow platforms still have significant shortcomings in operator extension and system architecture. On the one hand, the operator extension mechanisms of existing platforms are generally quite closed, often requiring adherence to specific interface specifications or even modification of existing code. The extension process is highly invasive, making it difficult to achieve flexible and fully customizable operator logic. On the other hand, existing solutions often lack a unified enhancement mechanism for AOP cross-cutting concerns such as logging, monitoring, security, and exception handling. Related functions often need to be implemented repeatedly in various operators, resulting in code redundancy and hindering unified maintenance and upgrades. Furthermore, the instance creation, dependency resolution, and lifecycle management of operators and related components lack a unified loosely coupled mechanism, easily leading to fragmented component management, high coupling, and difficulties in dynamic replacement and testing. For the loading process of multi-type and multi-level configurations, priority control and chained processing capabilities are generally lacking, making it difficult to ensure the orderly and reliable loading order of configurations and easily causing runtime errors due to incomplete dependencies.

[0005] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0006] To address the problems in related technologies, this invention proposes a design pattern-based DAG operator extension method and system to overcome the aforementioned technical problems in existing related technologies.

[0007] Therefore, the specific technical solution adopted by the present invention is as follows:

[0008] According to one aspect of the present invention, a design pattern-based DAG operator extension method is provided. This method includes: based on each original operator class in the DAG workflow to be executed, using the decorator pattern to associate and register business logic classes with the original operator classes, dynamically generating an operator class hierarchy including operator classes, proxy classes, and factory classes; using the factory pattern to determine the creation method of proxy instances corresponding to each original operator class, and using the dependency injection pattern to scan, instantiate, and register the factory classes in a container to obtain operator factory instances corresponding to each operator; and using the proxy pattern to execute the operators of the proxy classes. The process involves process control and aspect enhancement; the chain of responsibility pattern is used to load the operator configuration information required by each original operator class in an orderly manner; based on the DAG workflow to be executed, the dependencies and execution order between each original operator class are parsed, and the corresponding operator factory instance is obtained from the dependency injection container according to the execution order and the corresponding proxy instance is generated. The proxy instance is called to execute the corresponding operator, and the execution result of the current operator is written into the input of the subsequent operator until all operators in the workflow are executed; the execution results of each operator are summarized to obtain the target processing result, and the target processing result is returned as the output result of the DAG workflow to be executed.

[0009] According to another aspect of the present invention, a design pattern-based DAG operator extension system is also provided. This system includes: a design pattern layer, used to associate and register business logic classes with original operator classes based on each original operator class in the DAG workflow to be executed, using the decorator pattern to dynamically generate an operator class hierarchy including operator classes, proxy classes, and factory classes; using the factory pattern to determine the creation method of proxy instances corresponding to each original operator class, and using the dependency injection pattern to scan, instantiate, and register the factory classes in a container to obtain operator factory instances corresponding to each operator; and using the proxy pattern to control and switch the operator execution process of the proxy classes. The system is enhanced; and the chain of responsibility pattern is used to load the operator configuration information required by each original operator class in an orderly manner; the DAG operator execution engine layer is used to parse the dependency relationship and execution order between each original operator class according to the DAG workflow to be executed, obtain the corresponding operator factory instance from the dependency injection container according to the execution order, generate the corresponding proxy instance, call the proxy instance to execute the corresponding operator, and write the execution result of the current operator into the input of the subsequent operator, until all operators in the workflow are executed; the business application layer is used to summarize the execution results of each operator, obtain the target processing result, and return the target processing result as the output result of the DAG workflow to be executed.

[0010] According to a third aspect of the present invention, an electronic device is provided.

[0011] Furthermore, the electronic device includes: a processor; a system bus; a main memory communicatively connected to the processor via the system bus; and an auxiliary memory communicatively connected to the processor via the system bus. The main memory or auxiliary memory stores a computer program executable by the processor, which, when executed by the processor, implements the steps of the above-described method.

[0012] The beneficial effects of this invention are as follows:

[0013] (1) This invention constructs a unified technical solution for DAG operator extension by coordinating the design of decorator pattern, factory pattern, proxy pattern, dependency injection pattern and chain of responsibility pattern, so that operator registration, creation, enhancement, configuration loading and execution scheduling can be realized in the same system, thereby improving the overall architectural integrity, system scalability and engineering maintainability of DAG workflow platform.

[0014] (2) This invention associates business logic classes with original operator classes by applying decorators to the original operator classes and dynamically generates an operator class system of operator classes, proxy classes and factory classes. Without developers having to write complex registration code or modify the existing platform structure, the zero-intrusion extension of operators can be completed, thereby effectively reducing the problems of closed, highly intrusive and low development efficiency of operator extension in the prior art.

[0015] (3) The present invention uses the proxy pattern to uniformly control the execution process of the operator and sets up multiple interception points around the execution of the operator, so that cross-cutting concerns such as logging, monitoring, security, and exception handling can be injected into the execution process of different operators in a unified manner, avoiding the repeated implementation of related functions in each operator, reducing code redundancy, and improving the reusability, consistency and convenience of subsequent maintenance and upgrade of the enhanced logic.

[0016] (4) This invention determines the creation method of proxy instances through the factory pattern, and combines the dependency injection container to automatically scan, instantiate, register and uniformly manage the factory class, so that the creation, acquisition and dependency organization of operator-related components can be completed under loose coupling conditions, thereby effectively reducing the direct dependency relationship between components, improving the dynamic replacement capability of components, testing convenience and the overall modularity of the system.

[0017] (5) This invention uses a dependency injection container to perform unified lifecycle management of components and supports access and location by component name, so that the factory class and other related components in the system can be organized and scheduled in a unified manner, thereby improving the standardization and consistency of component management and enhancing the system's stable operation capability in complex business scenarios.

[0018] (6) The present invention loads the configuration information required by the operator in an orderly manner through the chain of responsibility pattern, and uses the loading method decorator to set the loading priority, construct the chain of responsibility loader class and execute the loading methods of each target in sequence, thereby ensuring that the global configuration, operator-level configuration and dependent resource configuration are loaded in a predetermined order, reducing runtime errors caused by uncontrolled configuration loading order or unready dependencies, and improving the reliability and flexibility of configuration management. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the process of operator registration using the decorator pattern in a DAG operator extension method based on design patterns according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the proxy pattern execution control in a DAG operator extension method based on design patterns according to an embodiment of the present invention;

[0022] Figure 3This is a schematic diagram of the dynamic component creation process of the factory pattern in a DAG operator extension method based on design patterns according to an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the management architecture of the dependency injection component in a DAG operator extension method based on design patterns according to an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of the configuration loading process of the chain of responsibility pattern in a DAG operator extension method based on design patterns according to an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram of the collaborative working architecture of five design patterns in a design pattern-based DAG operator extension method according to an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram of a DAG operator extension system based on a design pattern according to an embodiment of the present invention.

[0027] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0028] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0029] According to embodiments of the present invention, a method, system, and device for DAG operator extension based on design patterns are provided.

[0030] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figures 1-6As shown, according to an embodiment of the present invention, a design pattern-based DAG operator extension method is provided. This method includes: based on each original operator class in the DAG workflow to be executed, using the decorator pattern to associate and register business logic classes with the original operator classes, dynamically generating an operator class hierarchy including operator classes, proxy classes, and factory classes; using the factory pattern to determine the creation method of proxy instances corresponding to each original operator class, and using the dependency injection pattern to scan, instantiate, and register the factory classes in a container to obtain operator factory instances corresponding to each operator; using the proxy pattern to control the operator execution process and enhance aspects of the proxy class; and using the chain of responsibility pattern... The algorithm loads the operator configuration information required by each original operator class in an orderly manner to complete the operator registration, instance creation preparation, execution enhancement preparation, and configuration loading preparation for each original operator class. According to the DAG workflow to be executed, it parses the dependencies and execution order between each original operator class, retrieves the corresponding operator factory instance from the dependency injection container according to the execution order, generates the corresponding proxy instance, calls the proxy instance to execute the corresponding operator, and writes the execution result of the current operator into the input of the subsequent operator until all operators in the workflow are executed. The execution results of each operator are summarized to obtain the target processing result, and the target processing result is returned as the output result of the DAG workflow to be executed.

[0031] In one embodiment, the decorator pattern includes: obtaining a user-defined business logic class and a primitive operator class, and applying a decorator to the primitive operator class to trigger the decorator to receive the business logic class as a parameter; after the decorator is triggered, dynamically creating the operator class, proxy class, and factory class through several calls to the built-in metafunction, and injecting the association relationship layer by layer; and completing the static association establishment between the primitive operator class, operator class, proxy class, and factory class by returning the factory class to replace the primitive operator class, so as to complete the zero-intrusive registration of the operator.

[0032] In one embodiment, the dynamic creation of operator classes, proxy classes, and factory classes through several calls to built-in metafunctions, and the injection of relationships layer by layer, includes: First, by calling the built-in metafunction once, constructing an operator class with the same name as the original operator class, and injecting a business logic class and operator initialization method into the operator class so that the operator class carries the business logic; Second, by calling the built-in metafunction twice, constructing a proxy class based on the operator class, and injecting an operator class reference and proxy initialization method into the proxy class so that the proxy class creates a corresponding operator instance upon instantiation; Third, by calling the built-in metafunction three times, dynamically constructing a factory class based on the proxy class, and injecting proxy class attributes into the factory class so that the factory class can be used for the creation of subsequent proxy instances.

[0033] In one embodiment, the proxy pattern includes: predefining a proxy base class for encapsulating the operator execution flow, and setting several interception points in the proxy base class that can be overridden by subclasses or injected through decorators; when registering an operator with a decorator, generating a corresponding proxy class based on the original operator class, and making the proxy class inherit from the proxy base class and associate it with the corresponding operator class; when the user chooses to create a custom proxy subclass, injecting aspect logic into the interception point of the proxy class by overriding any one of the several interception points; when the user chooses to directly apply the decorator corresponding to the interception point to the operator class, automatically registering the function defined in the decorator to the interception point slot of the corresponding proxy class; when calling an operator, the execution engine obtains the proxy instance corresponding to the proxy class and calls the proxy instance to execute the corresponding operator, so that the proxy instance controls the operator execution process according to the execution order of the interception points.

[0034] In one embodiment, the interception points, in the order of execution, include: a pre-interception point for performing pre-checks or preparation work; a surround enable judgment interception point for determining whether to enable full surround enhancement; a surround start interception point for enabling resources or recording the context of entering the operator; an operator core logic execution interception point for calling the proxied original operator instance based on the proxy object to execute the operator's core business logic; a surround end interception point for ending surround enhancement after the operator's core business logic has been executed; a post-processing interception point for performing unified cleanup work; and a return and exception interception point for handling the return value when the operator executes successfully and handling exceptions when exceptions are thrown during execution.

[0035] In one embodiment, the factory pattern includes: when the original operator class is registered by the decorator, constructing a factory class corresponding to the original operator class and writing the corresponding proxy class into the proxy class attribute of the factory class; during the system initialization phase, instantiating the factory class and registering it to the dependency injection container to form operator factory instances corresponding to each operator; when the execution engine needs to execute the target operator, obtaining the corresponding operator factory instance from the dependency injection container according to the operator identifier of the target operator, and performing type validation on the operator factory instance; if the validated object type does not match the expectation, throwing an exception; creating a corresponding proxy instance from the operator factory instance and returning the proxy instance to the execution engine so that the execution engine can call the proxy instance to execute the target operator.

[0036] In one embodiment, creating a corresponding proxy instance from an operator factory instance includes: first, calling the operator creation method to create an operator object corresponding to the target operator, and injecting the configuration object or resource object that the operator depends on into the operator object in the operator creation method; then, creating a corresponding proxy instance based on the operator object, and writing the operator object into the proxy instance so that the proxy instance holds the proxied operator object; when accessing the proxy to obtain attributes of the operator factory instance, the proxy instance creation process is triggered as needed; when the operator factory instance sets a proxy cache, the proxy instance is created and cached on the first access, and the cached proxy instance is directly returned on subsequent accesses to achieve lazy loading of the proxy instance.

[0037] In one embodiment, the dependency injection pattern includes: establishing a dependency injection container; the dependency injection container is a globally unique container object used to store component instances in key-value pairs and supporting key-value or property access by component name; at system startup, by automatically scanning the modules to be loaded under a specified path, selecting subclasses of the loadable base class, instantiating the selected subclasses and registering them to the dependency injection container; allowing users to manually add components to the dependency injection container at any location; and the dependency injection container uniformly managing the lifecycle of the components.

[0038] In one embodiment, automatically scanning the modules to be loaded under a specified path to filter out subclasses of loadable base classes includes: recursively traversing program files under a preset scanning path, excluding program files containing preset directories and initialization files, retaining program files ending with preset file types as files to be scanned, and generating a list of files to be scanned; loading each file to be scanned by traversing the list of files to be scanned, and obtaining the target class corresponding to the file identifier; when the target class is a subclass of a loadable base class, adding the target class to the set of classes to be registered; instantiating the target classes belonging to the factory class in the set of classes to be registered, and writing them to the dependency injection container with the factory class name or a custom name as the key to form operator factory instances corresponding to each operator.

[0039] In one embodiment, the Chain of Responsibility pattern includes: marking target loading methods that need to be included in the configuration loading process by creating a loading method decorator, and setting loading priorities for each target loading method; during system initialization, scanning each target loading method in the loader class, extracting the corresponding loading method and loading priority to form a set of loading methods to be executed; wherein, the loader class is a class used to carry configuration loading logic and contains methods marked by the loading method decorator, and the target loading methods are methods marked by the loading method decorator and included in the configuration loading process; sorting the set of loading methods to be executed in ascending order according to loading priority, constructing a Chain of Responsibility loader class, and writing each sorted target loading method into the Chain of Responsibility loader class; executing loading sequentially according to the order of the target loading methods in the Chain of Responsibility loader class, so that global configuration, operator-level configuration, and dependent resource configuration are loaded into the corresponding operator or execution environment in a predetermined order.

[0040] In one embodiment, the set of methods to be executed is sorted in ascending order according to loading priority. Constructing the chain of responsibility loader class includes: arranging the target methods in ascending order according to their loading priority values; when several target methods have the same loading priority value, sorting them according to the lexicographical order of their method names; when several target methods have the same loading priority value and the same method name, sorting them according to the class definition order or class loading order to obtain a sequence of loaded methods; constructing the chain of responsibility loader class based on the sequence of loaded methods, and assigning an execution order identifier to each sorted target method in the chain of responsibility loader class; and storing a reference to the original loader instance in the chain of responsibility loader class so as to call the non-decorator methods in the original loader instance or access its context state when executing the target methods.

[0041] It should be noted that the core of this invention lies in the systematic and collaborative application of five design patterns in the specific problem domain of DAG operator extension, forming a complete lifecycle management mechanism from operator definition, registration, creation, enhancement, execution to configuration loading. Its collaborative architecture is as follows: Figure 6 As shown, it is divided into three layers: design pattern layer, DAG operator execution engine layer, and business application layer.

[0042] I. Design Pattern Layer:

[0043] This layer is composed of five design patterns working together to provide standardized and reusable technical support for the definition, creation, control, management, and configuration of operators, including:

[0044] 1) Decorator pattern (@OperatorDecorator) is used for zero-intrusion registration of operators and automatically builds a complete operator class hierarchy.

[0045] Specifically, the operator class hierarchy refers to three closely related classes dynamically generated through the `@OperatorDecorator` decorator: The original operator class (e.g., `MyOperator`): inheriting from `Operator`, containing placeholders or identifiers for business logic. The proxy class (`MyOperatorProxy`): inheriting from `OperatorProxy`, encapsulating the operator's execution control logic (e.g., AOP enhancements). The factory class (`MyOperatorProxyFactory`): inheriting from `ProxyFactory`, used to dynamically create proxy instances. The hierarchy is built during the operator definition phase, dynamically generated by the `@OperatorDecorator` decorator during class loading, establishing dependencies between them.

[0046] 2) Factory pattern (ProxyFactory) is used to dynamically create operator proxy instances, ensuring the type safety and consistency of instance creation.

[0047] Specifically, operator factory instances include: factory class instances, such as MyOperatorProxyFactory, which are automatically instantiated and registered by the dependency injection container when the system starts; and proxy instances, which are OperatorProxy subclass instances dynamically generated through the factory's proxy property, for the execution engine to call.

[0048] Specifically, factory instances are created and managed uniformly by the dependency injection container during the system initialization phase; proxy instances are generated by the factory on demand before the operator is executed.

[0049] 3) Dependency Injection Pattern (Inst Container): Used to achieve loosely coupled management and service location of components (such as factories).

[0050] Specifically, through the global container `insts`, combined with an automatic scanning mechanism, all factory classes are instantiated and registered in the container for the execution engine to retrieve on demand. The component management scope of the dependency injection pattern includes all factory classes, configuration classes, loaders, etc., that inherit from `ProxyFactory`.

[0051] 4) Operator Proxy pattern, used to provide unified execution control and AOP enhancement capabilities for operator execution.

[0052] Specifically, this invention defines seven interception points (before, use_around, start_around, end_around, after, after_return, after_exception) in the proxy pattern, supporting cross-cutting logic such as logging, monitoring, and exception handling before and after operator execution.

[0053] 5) Chain of Responsibility pattern (LoaderSlot & @LoadMethod) is used to ensure the orderly loading and correctness of multi-level and multi-type configurations and dependencies.

[0054] Specifically, operator configuration information includes environment variables, parameter settings, dependent resources, security policies, etc., required for operator execution. During operator initialization, this configuration information is loaded in priority order using a chain-of-responsibility pattern (e.g., global configuration is loaded first, then operator-level configuration), ensuring all dependencies are ready before operator execution. In this specific embodiment, the loading priority is set using the `@LoadMethod(sort)` decorator, and `LoaderSlot` dynamically constructs the execution chain, calling the methods sequentially.

[0055] II. DAG Operator Execution Engine Layer:

[0056] This layer is the central hub for operator scheduling and execution, and consists of actual program modules including the following core components: WorkflowParser, responsible for parsing user-defined DAG structures (such as JSON / YAML format), constructing operator dependency graphs, and generating executable execution plans; Scheduler, responsible for concurrently scheduling operators without dependencies based on operator dependencies, and managing execution queues and thread pools; Executor, responsible for actually invoking operators, obtaining factory instances through the dependency injection container, generating proxy instances, and calling their __call__ methods to execute operator logic; and ContextManager, responsible for maintaining the global context during workflow execution, including input data, intermediate results, and output data.

[0057] Specifically, the core responsibilities of the DAG operator execution engine layer include: workflow orchestration, which involves resolving the dependencies between operators and constructing an execution sequence graph based on the DAG definition; parallel scheduling, which involves concurrently scheduling and executing operators without dependencies to improve execution efficiency; result aggregation, which involves collecting the execution results of each operator as input or final output for subsequent operators; and interaction with the design pattern layer, whereby the execution engine obtains factory instances through the dependency injection container (insts), generates proxy instances, calls their __call__ methods to execute operator logic, and automatically triggers AOP interception points defined in the proxy pattern during execution.

[0058] Specifically, the code example for the call relationship is as follows:

[0059] #Examples of internal code for the execution engine

[0060] class Executor:

[0061] def execute_operator(self, operator_name: str, input_data: dict):

[0062] #1. Obtain a factory instance from the dependency injection container

[0063] factory = insts.get(operator_name, expected_type=ProxyFactory)

[0064] #2. Generating proxy instances via a factory

[0065] proxy = factory.proxy

[0066] #3. Call the proxy to execute the operator (automatically triggers AOP interception points)

[0067] result = proxy(input_data)

[0068] Return result.

[0069] III. Business Application Layer:

[0070] This layer comprises the specific business systems built by developers based on the framework of this invention, and consists of actual application modules. Typical scenarios include: users triggering a DAG workflow via a REST API or command-line interface; the business application inputs initial data and calls the execution engine's API to start the process; the engine executes operators sequentially or concurrently according to the DAG definition, and finally returns the processing result; during operator execution, logs, monitoring metrics, and exception information are automatically recorded, and users do not need to worry about the underlying enhancement logic.

[0071] Specifically, the program carriers of the business application layer include: API interface modules, such as api / workflow_api.py, which provide HTTP interfaces for front-end calls; business controllers, such as controllers / pipeline_controller.py, which are responsible for assembling workflow parameters and calling the execution engine; and custom operator modules, which are user-written business logic classes and operator classes, located in the operators / directory.

[0072] Specifically, the code example for the call relationship is as follows:

[0073] #Business Application Layer Code Examples

[0074] from engine import WorkflowEngine

[0075] class PipelineController:

[0076] def __init__(self):

[0077] self.engine = WorkflowEngine()

[0078] def run_pipeline(self,workflow_definition:dict,initial_data:dict):

[0079] #Invoke the execution engine to start the workflow

[0080] result = self.engine.run(workflow_definition, initial_data)

[0081] Return result.

[0082] It should also be noted that, in this invention, the complete call flow between the above three layers is as follows:

[0083] 1) The business application layer receives user requests, constructs workflow definitions and initial data, and calls the API of the DAG operator execution engine layer.

[0084] 2) The execution engine layer parses the workflow definition and determines the operator execution order. For each operator to be executed:

[0085] Obtain the corresponding factory instance through the dependency injection container (insts) of the design pattern layer;

[0086] Generate proxy instances using factory instances (proxy mode);

[0087] Call the proxy instance to execute the operator, which will automatically trigger AOP intercept points (logs, monitoring, etc.);

[0088] The proxy instance internally calls the business logic of the original operator class (generated by the decorator pattern).

[0089] 3) The execution engine layer collects the results of operator execution and continues to schedule subsequent operators until the workflow is completed.

[0090] 4) The execution engine layer returns the final result to the business application layer, which then returns it to the user.

[0091] Specifically, in this invention, the operator class system, namely the complete class structure of operator classes, proxy classes, and factory classes, is automatically generated by the decorator pattern during definition, and its determination time is the operator definition phase, i.e., when the code is loaded; the operator factory instance includes the factory class instance and its generated proxy instance, which is used to dynamically create and execute operators, and its determination time is the system initialization phase (factory instantiation) before operator execution (proxy instance generation); the operator configuration information, namely the environmental parameters, dependent resources, etc. required for operator operation, is loaded according to priority through the chain of responsibility pattern, and its determination time is the operator initialization phase, i.e., preparation before execution.

[0092] IV. Implementation of the Decorator Pattern in Operator Registration:

[0093] like Figure 1 As shown, the decorator pattern implementation of the present invention includes the following specific steps:

[0094] Step S101: Begin applying the decorator:

[0095] When defining a custom operator, developers first write the business logic class (inherited from Logic), then define the operator class (inherited from Operator), and apply the @OperatorDecorator decorator to the operator class, passing the business logic class as a parameter. This operation marks the start of the decorator pattern; all subsequent dynamic generation of classes is automatically triggered by the decorator. A partial code example is shown below:

[0096] “class MyLogic(Logic):

[0097] def __call__(self, operator: Operator) -> Operator: # Implementation of business logic

[0098] return operator

[0099] @OperatorDecorator(MyLogic)#Apply Decorator

[0100] class MyOperator(Operator):

[0101] "pass".

[0102] Step S102: Define the OperatorDecorator class:

[0103] OperatorDecorator is a decorator class whose constructor __init__ takes a logic_class parameter to store the business logic class passed in by the developer. This class must implement the __call__ method so that it is automatically invoked after the class definition is completed. A partial code example is shown below:

[0104] "class OperatorDecorator:

[0105] def __init__(self, logic_class: Type[Logic]):

[0106] self._logic_class = logic_class # Store the business logic class

[0107] def __call__(self, operator_class: Type[Operator]):

[0108] # Dynamically generate a three-tier class structure and return a factory class

[0109] ...”.

[0110] Step S103: The decorator calls the __call__ method:

[0111] When the Python interpreter executes the decorated operator class definition (i.e., `classMyOperator(Operator):pass`), it automatically calls the `__call__` method of the `OperatorDecorator` instance, passing the original operator class (`MyOperator`) as an argument. At this point, the decorator obtains the definition of the original operator class, preparing it for dynamically generating a new class.

[0112] Triggering timing explanation: @OperatorDecorator(MyLogic) is instantiated before the class definition. After the classMyOperator... statement is executed, the interpreter will pass the class object to the __call__ method of the decorator instance.

[0113] Step S104: Dynamically create a three-tier class structure:

[0114] Inside the __call__ method, three new classes are dynamically created by calling the built-in type metafunction three times, and the association relationship is injected layer by layer.

[0115] ① Create an operator class (operator_cls) and inject logic_class.

[0116] Specifically, the first call to `type` generates a new operator class (named after the original class name, such as `MyOperator`). This class inherits from the passed-in original operator class (`operator_class`) and injects the business logic class through the class attribute `logic_class`. Simultaneously, the helper function `create_init_method` generates a custom `__init__` method, passing `reload_operator_init` as a post-processing function; a partial code example is shown below:

[0117] "operator_cls = type(

[0118] operator_class.__name__, # Class name: MyOperator

[0119] (operator_class), #Base class: primitive operator class

[0120] {

[0121] 'logic_class': self._logic_class, # Inject the business logic class

[0122] '__init__':

[0123] create_init_method((operator_class,), reload_operator_init)

[0124] }

[0125] )".

[0126] Specifically, the implementation of `create_init_method` is as follows: This higher-order function generates an `__init__` method, ensures that all base classes are correctly initialized, and executes post-processing functions; a partial code example is shown below:

[0127] "def create_init_method(base_classes, reload_func):

[0128] def __init__(self):

[0129] # Call the __init__ function of all base classes in sequence

[0130] for base_class in base_classes:

[0131] if hasattr(base_class, '__init__'):

[0132] base_class.__init__(self)

[0133] # Execute the post-processing function, passing in the current instance.

[0134] reload_func(self)

[0135] return __init__.

[0136] Specifically, the implementation of `reload_operator_init` is as follows: This function is responsible for instantiating the business logic class and attaching it to the operator object; a partial code example is shown below:

[0137] "def reload_operator_init(self: Operator):

[0138] #self is a dynamically generated instance of operator_cls

[0139] #self.logic_class points to the MyLogic class defined by the developer.

[0140] self._logic = self.logic_class() # Create a business logic instance.

[0141] ② Create a proxy class (proxy_cls) and inject the operator class.

[0142] Specifically, the second call to `type` generates a new proxy class (the class name is the original class name plus "Proxy", such as `MyOperatorProxy`). This class inherits from the base proxy class `OperatorProxy` and injects the operator class generated in the previous step through the class attribute `operator_cls`. Similarly, `create_init_method` is used to generate the `__init__` method, and `reload_proxy_init` is passed in as the post-processing function; a partial code example is shown below:

[0143] "proxy_cls = type(

[0144] operator_cls.__name__ + 'Proxy', #Class name: MyOperatorProxy

[0145] (OperatorProxy), #Base class: Basic proxy class

[0146] {

[0147] 'operator_cls': operator_cls, # Injected operator class

[0148] '__init__':

[0149] create_init_method((OperatorProxy,), reload_proxy_init)

[0150] }

[0151] )".

[0152] Specifically, the implementation of `reload_proxy_init` involves two core initialization tasks: instantiating primitive operators and injecting AOP interceptors. A partial code example is shown below.

[0153] "def reload_proxy_init(self: OperatorProxy):

[0154] #1. Instantiating primitive operators

[0155] self._operator = self.operator_cls()

[0156] #operator_cls is the operator class generated in the previous step.

[0157] #2. Inject AOP interceptors (taking the before slot as an example, the same applies to other slots)

[0158] for func in OperatorProxy.before_decorator_slot:

[0159] self.before = GlobalDecorator(func).operator_proxy_chain(self)(self.before)

[0160] #..Handling of other slots (use_around, start_around, end_around, after, after_return, after_exception).

[0161] ③ Create a factory class (factory_cls) and inject the proxy class.

[0162] Specifically, the third call to `type` generates a new factory class (using the same name as the original class, such as `MyOperator`). This class inherits from both `LoadableBasic` and `ProxyFactory`, and injects the proxy class generated in the previous step through the `proxy_cls` class attribute. Ultimately, this factory class replaces the original `MyOperator` definition; a partial code example is shown below:

[0163] "factory_cls = type(

[0164] operator_class.__name__, #Class name: MyOperator

[0165] (LoadableBasic, ProxyFactory), #Base classes: Framework base classes and factory base classes

[0166] {'proxy_cls': proxy_cls} # Inject proxy class

[0167] )".

[0168] Step S105: Return to the factory class:

[0169] Specifically, the `__call__` method ultimately returns a dynamically generated factory class, `factory_cls`. Python binds this returned class to the original class name (i.e., `MyOperator` now points to the factory class). Developers will subsequently access the factory class through `MyOperator`, not the original simple operator class.

[0170] Step S106: Complete the non-intrusive registration:

[0171] Specifically, at this point, the decorator pattern in this invention has automatically completed the following tasks:

[0172] 1) An operator class (operator_cls) with business logic was created: This class inherits from the original operator class and holds the business logic class through the logic_class attribute. During instantiation, an instance of the business logic class is automatically created and attached to self._logic. The role of the operator class is to carry the specific business logic and is responsible for executing the core functionality of the operator, but it does not involve execution flow control or enhancement logic.

[0173] 2) A proxy class (proxy_cls) is created to control the execution flow: This class inherits from OperatorProxy and holds the operator class generated in the previous step through the operator_cls attribute. During instantiation, an instance of the operator class (i.e., the original operator object) is automatically created and AOP interceptors are injected. The proxy class's role is to control the entire operator execution flow, including inserting enhanced logic (such as logging, monitoring, exception handling, etc.) before and after the original operator call, achieving unified execution lifecycle management.

[0174] 3) A factory class (factory_cls) responsible for instantiating proxies is created: This class inherits from ProxyFactory and holds the proxy class through the proxy_cls property. The instantiated object of the factory class is the factory object used to generate proxy instances; it is instantiated and registered in the dependency injection container when the system starts. When an operator needs to be executed, the execution engine obtains the factory object from the container and then dynamically creates the proxy instance through its proxy property or create_operator method. The factory pattern ensures that the creation process of proxy instances is decoupled from the user and supports advanced features such as dependency injection and lazy loading.

[0175] 4) A static association was established among the three: through mutual references by class attributes logic_class, operator_cls, and proxy_cls, a clear hierarchical relationship and division of responsibilities were formed among the operator class, proxy class, and factory class, namely, the factory class holds the proxy class; the proxy class holds the operator class; and the operator class holds the business logic class.

[0176] Therefore, in this invention, developers only need to provide a business logic class and an empty operator class, and apply the `@OperatorDecorator` decorator. Without writing any registration code or implementing complex interfaces, the dynamic generation and association of the aforementioned three-tier class structure can be automatically completed, achieving truly zero-intrusion operator extension. The subsequent framework obtains the factory object through a dependency injection container, then generates proxy instances, and the proxy uniformly manages the execution and enhancement of operators. The entire process is completely transparent to the developer.

[0177] V. Implementation of the Proxy Pattern in Execution Control:

[0178] like Figure 2 As shown, the proxy mode implementation of the present invention includes the following specific steps:

[0179] Step S201: Define an abstract proxy base class with seven interception points. That is, the system predefines an abstract proxy base class OperatorProxy, which encapsulates the entire process of operator execution and exposes seven interception point methods that can be overridden by subclasses or injected through decorators.

[0180] Specifically, these seven interception points are executed in the following order:

[0181] ① `before` pre-intercept point: Called before the core logic of the operator is executed, used to perform pre-checks or preparatory work (such as parameter validation, permission verification). This method returns a boolean value. If it returns `False`, the surrounding logic is skipped and the original operator is executed directly; if it returns `True`, the surrounding logic continues to be executed.

[0182] ② `use_around` Enabling Interception Point: Determines whether to enable full surround enhancement. This method is called after `before` and is used to dynamically control the activation state of the surround logic to adapt to the enhancement needs of different scenarios.

[0183] ③start_around: marks the start of the around enhancement and is typically used to start resources (such as timers or transactions) or record the context of entering an operator.

[0184] ④ Execution of core operator logic: After start_around, the proxy object will call the __call__ method of the proxied original operator instance to execute the core business logic of the operator.

[0185] ⑤ end_around: Called after the core logic of the operator has been executed, to end the around enhancement, such as stopping the timer, committing the transaction, or cleaning up resources.

[0186] ⑥ after interception point: Regardless of whether the operator succeeds or fails, it will be executed after end_around, and is used to perform unified cleanup work (such as releasing the connection).

[0187] ⑦ after_return / after_exception return and exception interception points: These two interception points are executed mutually exclusively. If the operator executes successfully, after_return is called to handle the return value; if an exception is thrown during execution, after_exception is called to handle the exception (such as logging the error or converting the exception).

[0188] Step S202: Construct a dynamic generation mechanism for proxy classes

[0189] The proxy class is not written manually by the developer, but is dynamically generated during operator registration using the decorator pattern described earlier. When a developer decorates a custom operator class with `@OperatorDecorator`, the system automatically creates a corresponding proxy class (e.g., `MyOperatorProxy`), which inherits from `OperatorProxy` and associates the original operator class with it. This dynamic generation process ensures that each operator has its own dedicated proxy class, which can inherit from or override the seven interception methods in the base class, thereby achieving targeted enhancements.

[0190] Step S203: Define the injection method for aspect enhancement. The system supports two methods to inject aspect logic into the interceptors of the proxy class: ① Inheritance overriding: Developers can create custom proxy subclasses and directly override any one of the seven interceptor methods to achieve customized behavior. ② Decorator injection: The system provides decorators corresponding to the interceptors, such as @Before, @After, and @Around. Developers can directly apply these decorators to the operator subclasses, and the framework will automatically register the functions defined in the decorators to the corresponding interceptor slots of the proxy class. Some code examples are shown below:

[0191] "@EndAround(online_end_around) # Injects the online_end_around function into the end_around slot"

[0192] @StartAround(online_start_around) # Injects the online_start_around function into the start_around slot

[0193] @OperatorDecorator(logic_class=OnlineLogic)

[0194] class OnlineOperator(Operator):

[0195] "pass".

[0196] This approach achieves complete decoupling between logic and operators, enhancing code reusability and maintainability.

[0197] Step S204: Instantiation and execution control of the proxy object. At runtime, the proxy object is not directly created by the developer. Instead, it obtains the corresponding proxy factory (automatically generated by the decorator pattern) from the global component repository `insts` through the dependency injection container, and then obtains the proxy instance through the factory's `proxy` property. When the workflow engine needs to execute a certain operator, it actually calls the `__call__` method of the proxy object. This method strictly follows the order of the seven interception points mentioned above. The core process is as follows:

[0198] ① Call before() to perform a pre-check; if it returns False, skip the wrap logic and directly execute the primitive operator self._operator().

[0199] ② If before() returns True, then continue to check use_around(); if around is enabled, then execute start_around(), the primitive operator self._operator(), and end_around() in sequence; if around is not enabled, then execute the primitive operator directly.

[0200] ③ After executing the original operator, call after() for post-processing.

[0201] ④ Based on the execution result: if it is normal, call after_return(); if an exception occurs, catch the exception and call after_exception(exception).

[0202] It should be noted that existing AOP implementations (such as Spring AOP and AspectJ) are typically based on dynamic proxies or bytecode weaving, providing limited interception points (such as @Before, @After, and @Around) before and after method calls. These often require additional configuration of pointcut expressions, resulting in some coupling with business logic. This invention, in the specific scenario of DAG operator extension, designs seven semantically clear interception points tailored to the characteristics of operator execution. These points not only cover conventional pre- and post-processing but also refine the control over enabling around logic, the phased management of around start and end points, and the separation of return values ​​and exceptions, providing richer extension dimensions for fine-grained control at the operator level.

[0203] Furthermore, the proxy pattern of this invention does not exist in isolation, but rather works closely with the decorator pattern, factory pattern, and dependency injection container. Specifically: its collaboration with the decorator pattern means that the proxy class is automatically generated during operator registration, eliminating the need for developers to manually write proxy classes and achieving zero-intrusion enhancement. Its collaboration with the factory pattern means that proxy instances are created through the factory pattern, ensuring type safety and consistent instance management. Its collaboration with the dependency injection container means that the proxy factory is automatically scanned and registered in the global container, allowing any module that needs to execute operators to obtain the proxy object through service location, achieving loose coupling between components. This proxy mechanism based on the collaboration of multiple design patterns makes adding logging, monitoring, and security functions to all operators in a DAG workflow system exceptionally simple, greatly improving the system's scalability and maintainability, and demonstrating significant substantial features and advancements.

[0204] VI. Implementation of the Factory Pattern in Component Creation:

[0205] like Figure 3 As shown, the factory pattern implementation of the present invention includes the following specific steps:

[0206] Step S301: The application of the decorator triggers the automatic creation of the factory class. That is, when defining the operator, the developer uses the @OperatorDecorator decorator to mark the custom operator class.

[0207] Specifically, this decorator automatically generates a three-tier class structure internally using metaprogramming techniques (dynamically creating classes using type): an operator class (such as MyOperator), inheriting from Operator, containing business logic; a proxy class (such as MyOperatorProxy), inheriting from OperatorProxy, used to control the lifecycle of operator execution (see Section 4.3); and a factory class (such as MyOperatorProxyFactory), inheriting from the base class ProxyFactory provided by the framework, and assigning the type reference of the proxy class to its class attribute proxy_cls. Some example code is shown below:

[0208] "@OperatorDecorator(MyLogic)

[0209] class MyOperator(Operator):

[0210] pass

[0211] After the decorator is executed, it is equivalent to creating the following three classes:

[0212] class MyOperator(Operator):...# Operator class

[0213] class MyOperatorProxy(OperatorProxy): # Proxy class

[0214] operator_cls = MyOperator # Associated operator class

[0215] class MyOperatorProxyFactory(ProxyFactory): # Factory class

[0216] proxy_cls = MyOperatorProxy # Specifies the proxy class.

[0217] Step S302: Instantiate the factory class and automatically register it to the global container insts.

[0218] Specifically, during system startup, a package scanning mechanism (see the `scanner` function in Section 4.5) automatically discovers all classes inheriting from `LoadableBasic`, including all dynamically generated factory classes. Subsequently, during the configuration initialization phase, the `_factory_build` method is called to instantiate all factory classes and store the instances in the global dependency container `insts` using the class name (or a custom name) as the key. Some sample code is shown below:

[0219] "def _factory_build(self) -> List[ProxyFactory]:

[0220] factories = [cls() for cls in classes if issubclass(cls,ProxyFactory)]

[0221] insts.update({factory.__class__.__name__: factory for factory infactories})

[0222] "return to factories".

[0223] At this point, each operator corresponds to a factory singleton, which is stored in insts for subsequent service location.

[0224] Step S303: The client requests a factory instance through the container.

[0225] Specifically, when the workflow engine needs to execute a certain operator, it retrieves the corresponding factory instance from insts based on the operator configuration (such as the value of the configuration item operator_class being "MyOperator"): factory = insts.get("MyOperator"); where insts is a custom dictionary container that supports attribute access (__getattr__), and here it returns the MyOperatorProxyFactory singleton.

[0226] Step S304, Type Safety Check, is a type verification mechanism provided by the framework to ensure that the obtained factory conforms to the expected type.

[0227] Specifically, the expected base class can be specified when retrieving the data, and the container will perform an isinstance check internally; some example code is shown below:

[0228] "factory = insts.get("MyOperator", expected_type=ProxyFactory)

[0229] if factory is None:

[0230] raise TypeError("Factory for MyOperator is not a validProxyFactory")".

[0231] If the actual object type does not match the expectation, insts.get returns None or throws an exception (depending on the implementation). This mechanism prevents subsequent runtime exceptions caused by type errors, ensuring type safety.

[0232] Step S305: Dynamically create agent instances in the factory.

[0233] Specifically, the factory class inherits from ProxyFactory, and its core responsibility is to generate proxy instances. The following is a partial example code of the ProxyFactory base class definition:

[0234] class ProxyFactory:

[0235] proxy_cls: Type[OperatorProxy] = OperatorProxy # Overridden by subclasses

[0236] def create_operator(self) -> Operator:

[0237] # Creates an operator instance, which can be overridden by subclasses to implement dependency injection.

[0238] # Operators are created using the parameterless constructor by default.

[0239] return self.proxy_cls.operator_cls() if hasattr(self.proxy_cls, 'operator_cls') else None

[0240] @property

[0241] def proxy(self) -> OperatorProxy:

[0242] #1. Creating an operator instance

[0243] operator = self.create_operator()

[0244] #2. Create a proxy instance and inject the operator into the proxy.

[0245] proxy_instance = self.proxy_cls(operator)

[0246] #3. Perform additional initialization of the proxy, such as setting interceptors.

[0247] "return proxy_instance".

[0248] Specifically, in `MyOperatorProxyFactory`, its `proxy_cls` already points to `MyOperatorProxy`. The default `create_operator` calls the parameterless constructor of `MyOperator`. If the operator needs to depend on external components (such as configuration or database connection), developers can override the `create_operator` method, obtain the dependencies from the `insts` container, and explicitly pass them in; some example code is shown below:

[0249] "class MyOperatorProxyFactory(ProxyFactory):

[0250] proxy_cls = MyOperatorProxy

[0251] def create_operator(self) -> Operator:

[0252] # Dependency Injection Example: Retrieving a Configuration Object from a Container

[0253] config = insts.get("DatabaseConfig")

[0254] `return MyOperator(config)` # Inject via constructor.

[0255] This process implements dependency injection initialization: the operator's dependencies are obtained and passed in from the container by the factory, rather than being hard-coded inside the operator or searched out by itself, thus reducing coupling.

[0256] Step S306: Lazy loading mechanism and cache management.

[0257] Specifically, the factory pattern supports two granularities of caching: factory instance caching, where, as mentioned above, all factory instances are created and cached in `insts` once at system startup (this is the default behavior, avoiding the overhead of repeated instantiation); and optional lazy-loaded proxy instance caching, where by default, a new proxy and operator instance is created each time `factory.proxy` is accessed, suitable for stateless operators. For stateful operators or those with high creation overhead, proxy instance caching can be implemented within the factory to achieve lazy loading, meaning the instance is created and cached on the first access, and subsequent accesses directly return the cached instance. Some sample code is shown below:

[0258] "class CachedProxyFactory(ProxyFactory):

[0259] def __init__(self):

[0260] super().__init__()

[0261] self._proxy_cache = None

[0262] @property

[0263] def proxy(self):

[0264] if self._proxy_cache is None:

[0265] self._proxy_cache = super().proxy # Initial creation and caching of the proxy

[0266] return self._proxy_cache".

[0267] Lazy loading refers to delaying the creation of proxy instances until the first actual use, controlled by cache flags. Developers can decide whether to enable caching and the caching strategy based on operator characteristics, such as thread-local caching, weak reference caching, etc.; the framework itself does not enforce caching, but provides extension points.

[0268] Step S307: Return the proxy instance to the client.

[0269] Specifically, the client ultimately obtains a fully initialized proxy instance with dependencies injected via `factory.proxy`. This proxy encapsulates the specific operators and is configured with AOP intercept points (see Section 4.3). The client can directly call the proxy() function to execute operators, and the proxy will execute control logic according to the predefined lifecycle methods such as before / around / after.

[0270] VII. Implementation of Dependency Injection Pattern in Component Management:

[0271] like Figure 4 As shown, the dependency injection implementation of the present invention includes the following specific steps:

[0272] Step S401: Define a global dependency container, i.e., create a globally unique dependency container class Inst. This class inherits from Python's built-in dict and provides attribute access capabilities, allowing components to be accessed directly by name. Some example code is shown below:

[0273] “class Inst(dict, Generic[K, V]):

[0274] def __init__(self, *args, **kwargs) -> None:

[0275] super().__init__(*args, **kwargs)

[0276] def __getattr__(self, name: str) -> Any:

[0277] If the name is not in self.keys():

[0278] Raise AttributeError(f"Component '{name}' is not registered")

[0279] return self[name].

[0280] The container stores component instances in key-value pairs, and the __getattr__ method supports access via insts.component name, simplifying the code for service location.

[0281] Step S402, Automatic Scanning and Component Registration, refers to the process where, upon system startup, the system scans modules under a specified path, automatically discovers classes that meet the criteria, instantiates them, and registers them with the container. Specific sub-steps include:

[0282] ① Define the scan path and filtering rules, i.e., set the basic scan directory path_dir, and use recursive traversal to collect all Python files. Filtering rules include: excluding directories containing 'framework' or 'script' in the path to avoid scanning framework or script files; excluding files with the filename '__init__.py'; and keeping only files ending with .py.

[0283] ② Collect the files to be scanned, i.e., use list comprehension to quickly generate a list of files to be scanned; some sample code is as follows:

[0284] “py_files = [

[0285] (r, filename)

[0286] for r, _, filenames in os.walk(path_dir)

[0287] if 'framework' not in r and 'script' not in r

[0288] for filename in filenames

[0289] if filename != '__init__.py' and filename.endswith('.py')

[0290] ]".

[0291] ③ Dynamically load modules and filter classes: This involves iterating through `py_files`, loading modules using `importlib.machinery.SourceFileLoader`, and retrieving classes from the modules whose names match the filenames (by convention, the filename is the class name). It then checks if the class is a subclass of `LoadableBasic`; if so, it adds it to the registration list `classes`. Some sample code is shown below:

[0292] "for r, filename in py_files:

[0293] try:

[0294] module = importlib.machinery.SourceFileLoader(

[0295] filename.replace('.py', ''), os.path.join(r, filename)

[0296] ).load_module()

[0297] class_imported = getattr(module, filename.replace('.py', ''),None)

[0298] if class_imported is None:

[0299] continue

[0300] if issubclass(class_imported, LoadableBasic):

[0301] classes.append(class_imported)

[0302] except Exception:

[0303] logging.error(traceback.format_exc())".

[0304] ④ Instantiate and register with the container. This involves iterating through classes (e.g., the `_factory_build` method), instantiating classes that inherit from `ProxyFactory`, and storing the instances in the `insts` container using the class name or a custom name as the key. Some sample code is shown below:

[0305] "def _factory_build(self) -> List[ProxyFactory]:

[0306] factories = [cls() for cls in classes if issubclass(cls,ProxyFactory)]

[0307] insts.update({factory.__class__.__name__: factory for factory infactories})

[0308] "return to factories".

[0309] At this point, the container contains all the automatically discovered factory components.

[0310] Step S403: Manually register components. In addition to automatic scanning, the framework allows developers to manually add components to the container at any location, such as registering classes under non-standard paths or overriding default implementations. Example code: "insts['custom_operator']=CustomOperatorFactory()".

[0311] Specifically, manually registered components have higher priority than automatically scanned components and can be overridden as needed to meet the requirements of flexible expansion.

[0312] Step S404, Dependency Resolution and Automatic Injection, means that when an object is created, the container automatically resolves the dependencies declared in its constructor or properties and finds the corresponding instance in the container to complete the injection. The specific implementation includes the following sub-steps:

[0313] ① Dependency declaration, that is, a component can declare dependencies through constructor parameter type annotations, as shown in the following example code:

[0314] "class Worker:

[0315] def __init__(self, logger: Logger, config: Config):

[0316] self.logger = logger

[0317] self.config = config.

[0318] ② Dependency resolution: When the container instantiates a Worker, it checks the parameter types of its constructor, retrieves the registered Logger and Config instances from `insts`, and passes them to the constructor. If the dependency does not exist, an explicit exception is thrown.

[0319] ③ Circular dependency detection: The container maintains a collection of components under creation. When resolving dependencies, if it finds that a component is already being created (i.e., A depends on B and B depends on A), it is considered a circular dependency and an error is raised to prevent deadlock. This can be implemented using a three-level caching mechanism to store completed instances, semi-finished instances, and factory methods, thus safely handling circular dependencies. For example, it can be exposed in advance through a proxy object.

[0320] Step S405, Service Location, means that the component user does not need to directly hold the dependency; they only need to obtain the target instance through the container. Two methods are supported: key-value access, e.g., proxy_factory=insts.get('MyOperator'); and property access, e.g., proxy_factory=insts.MyOperator. The instance returned by the container can be used directly without needing to concern themselves with its creation process and dependencies.

[0321] Step S406, Lifecycle Management, refers to the container's unified management of the component's lifecycle, including scope control, initialization callbacks, and destruction callbacks:

[0322] ① Singleton and prototype support: By default, all components are singletons, created once when the container starts and reused subsequently. If prototypes are required (a new instance is created each time it is retrieved), they can be specified via metadata tags such as `@Scope('prototype')` or configuration. The factory class `ProxyFactory` returns a singleton proxy by default, but it can be overridden by subclasses to implement prototypes.

[0323] ② Initialization callback: If a component needs to execute specific initialization logic after dependency injection is completed, it can implement the InitializingBean interface, such as defining the after_properties_set() method; the container will automatically call this method after instantiation and dependency injection.

[0324] ③ Destruction callback: If a component holds resources that need to be released, it can implement the DisposableBean interface (e.g., define the destroy() method). When the application is closed, the container will iterate through all singleton components and call their destruction methods to ensure that the resources are properly reclaimed.

[0325] Step S407: Achieve loosely coupled component relationships. Through the dependency injection mechanism described above, components no longer directly create dependencies using `new` or hard-coded references; instead, the container handles assembly. Specifically, components only depend on abstract interfaces or types, without needing to concern themselves with concrete implementation classes; component configuration, replacement, and testing can all be completed by modifying container registration items, without modifying the component's internal code; the overall system modularity is high, and maintainability is significantly improved.

[0326] 8. Implementation of the Chain of Responsibility Pattern in Configuration Loading:

[0327] like Figure 5 As shown, the implementation of the chain of responsibility pattern of the present invention includes the following specific steps:

[0328] Step S501: Define the load method decorator @LoadMethod.

[0329] Specifically, a LoadMethod decorator is created to mark the method that needs to be included in the configuration loading chain and to specify its execution priority. This decorator accepts an integer parameter `sort` (default value is 1), with smaller values ​​indicating higher priority and earlier execution. Internally, the decorator encapsulates the original function being decorated into a LoadFunction object, which stores the original function reference and priority value. Subsequent ChainSetter decorators then add the ability to bind to the loader instance.

[0330] Step S502: Scan the loader class and collect the methods marked with @LoadMethod.

[0331] Specifically, during system initialization, all loader classes (or instances) inheriting from LoaderBasic are scanned using reflection, all their methods are traversed, and methods decorated with @LoadMethod are identified. Each identified method is encapsulated as an independent LoadFunction object, and its corresponding loader class or instance is recorded, forming a set of methods to be executed.

[0332] Step S503: Sort the loading methods in ascending order by priority using the sort function.

[0333] Specifically, all LoadFunction objects are sorted in ascending order according to their sort property values. If two methods have the same sort value, a stable sorting strategy is used: first, sort by lexicographical order of method names; if method names are also the same, sort by the order in which they appear in the code (or the class loading order), ensuring that the execution order is determined and predictable each time. The sorted sequence is the execution order of configuration loading.

[0334] Step S504: Construct the chain of responsibility loader class LoaderSlot.

[0335] Specifically, the LoaderSlot class is created, and its core static method loader_set is responsible for dynamically constructing the responsibility chain for configuration loading. loader_set receives a raw loader instance and performs the following operations:

[0336] ① Extract all methods decorated with @LoadMethod (i.e., a collection of LoadFunction objects) from the class to which the loader belongs and its parent classes.

[0337] ② Sort the above methods according to the rules in step S503.

[0338] ③ Dynamically create a new class that inherits from the original classes LoaderSlot and loader, and generate the new class name based on the original class name.

[0339] ④ In the new class, dynamically add a method named load_func_N (N increments from 1, corresponding to the execution order) to each sorted LoadFunction object. These methods will internally call the original LoadFunction object.

[0340] ⑤ Store a reference to the original loader instance in the new class, saved by an attribute (e.g., an attribute named after the original class) so that non-decorator methods in the original class can be called directly when necessary.

[0341] ⑥ Return this newly created class (or its instantiated object) as the enhanced loader.

[0342] Step S505: Dynamic link construction and instance binding.

[0343] Specifically, during the construction of the new class `LoaderSlot.loader_set`, the `ChainSetter` decorator adds an `owner_chain` method to each `LoadFunction` object. The `owner_chain` method takes a loader instance parameter and binds it to the `LoadFunction` (e.g., setting the `_owner` property), allowing subsequent calls to the function to access the context of the current loader instance through the `owner` property. This mechanism ensures that the decorated method can obtain the state of the loader instance or call its other methods during execution.

[0344] Step S506: Execute the loading method sequentially.

[0345] Specifically, a `load` method is defined in the newly constructed class. This method executes all `load_func_N` methods in sequence. Partial example code is shown below:

[0346] "def load(self):

[0347] if self.end_sort == 0:

[0348] return self

[0349] for i in range(1, self.end_sort + 1):

[0350] func = getattr(self, 'load_func_' + str(i))

[0351] func.owner_chain(self)() # Binds the current instance to a function and calls it.

[0352] "return self".

[0353] Where end_sort represents the total number of methods after sorting. Each load_func_N internally calls the original decorated method, and this method can access the loader instance through the owner attribute, thereby reading configuration data or performing dependency operations.

[0354] Step S507, Error Isolation and Recovery Mechanism.

[0355] Specifically, an exception handling mechanism is used to isolate errors during the execution of each `load_func_N`. If a loading method throws an exception, the system logs the error, such as outputting the stack trace using `logging.error`, but does not interrupt the execution of subsequent methods. This design ensures that the failure to load a single configuration item will not affect the loading of other configuration items. At the same time, error recovery interfaces can be reserved: for example, after catching an exception, an attempt can be made to set default values ​​or mark the system as failed, but at least the loading chain is guaranteed to continue. After all methods have been executed, the system can decide whether to mark the entire system as a loading failure or a partial success based on the error log.

[0356] Step S508: All configurations have been loaded.

[0357] Specifically, once all `load_func_N` calls have finished, the `load` method returns the current loader instance, indicating the end of the configuration loading process. At this point, all configuration items have been loaded in the predetermined priority order, and dependencies have been correctly satisfied. The returned instance can be used for subsequent operator initialization or other components.

[0358] It should be noted that in practical applications, multiple loader classes inheriting from LoaderBasic may define loading methods with the same name, for example, all defining load_database_config. This invention ensures correct invocation through the following mechanism: each method, when decorated with @LoadMethod, is independently encapsulated as a LoadFunction object, carrying its original method reference and priority; when constructing a new class in LoaderSlot.loader_set, all decorated methods of all loader instances are collected and uniformly sorted. Methods with the same name are treated as different objects due to their different origins, each possessing an independent sort value; the dynamically created new class renames these methods to load_func_1, load_func_2, etc., and the original method name is no longer used as an invocation identifier. Therefore, methods with the same name will be assigned to different execution locations, uniquely identified by load_func_N; if a loader needs to call the original non-decorated method of another loader, it can directly call it through the original instance reference saved in the new class (e.g., an attribute named after the original class name), but this is not within the scope of the configuration loading chain.

[0359] Specifically, the priority sorting rules and conflict handling in this invention are as follows: n in @LoadMethod(sort=n) is an integer. The smaller the value, the higher the priority and the earlier the execution order. When sort is not specified, the default value is 1. If multiple methods have the same sort value, a stable sorting is adopted: sorting is prioritized according to the lexicographical order of the method names. If the method names are the same, they are sorted according to the order in which they appear in the class definition or the class loading order to ensure determinism. Methods within the same loader are sorted by sort, and methods from different loaders are also sorted uniformly by sort, thereby realizing global priority control across loaders.

[0360] Through the above steps, this invention realizes a sortable, isolating, and recoverable configuration loading mechanism based on the chain of responsibility pattern, which significantly improves the reliability and flexibility of configuration management in the DAG operator system.

[0361] like Figure 7 As shown, according to another embodiment of the present invention, a design pattern-based DAG operator extension system is also provided, which includes:

[0362] The design pattern layer is used to associate and register business logic classes with original operator classes based on the original operator classes in the DAG workflow to be executed, dynamically generating an operator class hierarchy including operator classes, proxy classes, and factory classes; the factory pattern is used to determine the creation method of proxy instances corresponding to each original operator class, and the dependency injection pattern is used to scan, instantiate, and register the factory class with the container to obtain the operator factory instance corresponding to each operator; the proxy pattern is used to control the operator execution process and aspect enhancement of the proxy class; and the chain of responsibility pattern is used to load the operator configuration information required by each original operator class in an orderly manner to complete the operator registration, instance creation preparation, execution enhancement preparation, and configuration loading preparation corresponding to each original operator class.

[0363] The DAG operator execution engine layer is used to parse the dependency relationships and execution order between each original operator class according to the DAG workflow to be executed, obtain the corresponding operator factory instance from the dependency injection container according to the execution order and generate the corresponding proxy instance, call the proxy instance to execute the corresponding operator, and write the execution result of the current operator into the input of the subsequent operator until all operators of the workflow are executed.

[0364] The business application layer is used to summarize the execution results of each operator, obtain the target processing result, and return the target processing result as the output result of the DAG workflow to be executed.

[0365] In one embodiment, an electronic device is provided. For example... Figure 8 As shown, the electronic device includes a processor, a system bus, main memory, and auxiliary memory. The processor communicates with both the main memory and the auxiliary memory via the system bus.

[0366] The processor provides computational and control capabilities. Main memory, or internal memory, provides the environment for the computer program to run. Secondary memory is a non-volatile storage medium that stores the operating system and the computer program. When the processor executes the computer program, it implements the steps described in the above embodiments.

[0367] Those skilled in the art will understand that Figure 8 The structure shown is only a block diagram of a part of the structure related to the present invention and does not constitute a limitation. The specific electronic device may include more or fewer components, or have different component arrangements.

[0368] In addition, the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps in the above embodiments.

[0369] The computer-readable storage medium includes main memory or secondary memory. The main memory may be random access memory (RAM), and the secondary memory may be a non-volatile storage medium such as read-only memory (ROM), flash memory, hard disk, or optical storage.

[0370] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware.

[0371] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extending DAG operators based on design patterns, characterized in that, include: Based on the original operator classes in the DAG workflow to be executed, the decorator pattern is used to associate and register the business logic classes with the original operator classes, and dynamically generate an operator class system including operator classes, proxy classes and factory classes. The factory pattern is used to determine the creation method of the proxy instance corresponding to each original operator class, and the dependency injection pattern is used to scan, instantiate and register the factory class with the container to obtain the operator factory instance corresponding to each operator. The proxy pattern is used to control the operator execution process and enhance aspects of the proxy class; and the chain of responsibility pattern is used to load the operator configuration information required by each original operator class in an orderly manner. Based on the DAG workflow to be executed, the dependencies and execution order between each original operator class are parsed. The corresponding operator factory instance is obtained from the dependency injection container according to the execution order and a corresponding proxy instance is generated. The proxy instance is called to execute the corresponding operator, and the execution result of the current operator is written into the input of the subsequent operator until all operators of the workflow are executed. The execution results of each operator are summarized to obtain the target processing result, and the target processing result is returned as the output result of the DAG workflow to be executed.

2. The DAG operator extension method based on design patterns according to claim 1, characterized in that, The decorator pattern includes: Obtain the user-defined business logic class and the original operator class, and apply a decorator to the original operator class to trigger the decorator to receive the business logic class as a parameter; After the decorator is triggered, the operator class, proxy class and factory class are dynamically created by calling the built-in metafunction several times, and the association relationship is injected layer by layer. By returning the factory class to replace the original operator class, the static association between the original operator class, operator class, proxy class, and factory class is established, thus achieving zero-intrusive operator registration.

3. The DAG operator extension method based on design patterns according to claim 2, characterized in that, The method of dynamically creating operator classes, proxy classes, and factory classes through several calls to built-in metafunctions, and injecting association relationships layer by layer, includes: By calling the built-in metafunction once, an operator class with the same name as the original operator class is constructed based on the original operator class, and a business logic class and operator initialization method are injected into the operator class so that the operator class can carry business logic; By calling the built-in metafunction twice, a proxy class is constructed based on the operator class, and the operator class reference and proxy initialization method are injected into the proxy class so that the proxy class creates the corresponding operator instance when it is instantiated. By calling the built-in metafunction three times, a factory class is dynamically constructed based on the proxy class, and the proxy class attribute is injected into the factory class so that the factory class can be used for the creation of subsequent proxy instances.

4. The DAG operator extension method based on design patterns according to claim 1, characterized in that, The proxy mode includes: A proxy base class is predefined to encapsulate the operator execution process, and several interception points that can be overridden by subclasses or injected through decorators are set in the proxy base class; When registering an operator in a decorator, a corresponding proxy class is generated based on the original operator class, and the proxy class inherits from the proxy base class and is associated with the corresponding operator class. When a user chooses to create a custom proxy subclass, aspect logic can be injected into the proxy class's interception point by overriding any one of several interception points. When a user chooses to apply the decorator corresponding to the intercept point directly to the operator class, the function defined in the decorator is automatically registered to the intercept point slot of the corresponding proxy class. When an operator is invoked, the execution engine obtains the proxy instance corresponding to the proxy class and invokes the proxy instance to execute the corresponding operator, so that the proxy instance controls the execution process of the operator according to the execution order of the interception points.

5. The DAG operator extension method based on design patterns according to claim 1, characterized in that, The factory pattern includes: When the original operator class is registered by the decorator, a factory class corresponding to the original operator class is constructed, and the corresponding proxy class is written into the proxy class attribute of the factory class; During the system initialization phase, the factory class is instantiated and registered to the dependency injection container to form operator factory instances corresponding to each operator; When the execution engine needs to execute a target operator, it retrieves the corresponding operator factory instance from the dependency injection container based on the operator identifier of the target operator, and performs type validation on the operator factory instance; if the validated object type does not match the expectation, an exception is thrown. The operator factory instance creates a corresponding proxy instance and returns the proxy instance to the execution engine so that the execution engine can call the proxy instance to execute the target operator.

6. The DAG operator extension method based on design patterns according to claim 1, characterized in that, The dependency injection patterns include: Establish a dependency injection container; the dependency injection container is a globally unique container object used to store component instances in key-value pairs and supports key-value or property access by component name; When the system starts, it automatically scans the modules to be loaded under the specified path, filters out the subclasses of the base class that can be loaded, instantiates the filtered subclasses and registers them to the dependency injection container; at the same time, it allows users to manually add components to the dependency injection container at any location. The dependency injection container provides unified management of the component lifecycle.

7. The DAG operator extension method based on design patterns according to claim 6, characterized in that, The step of automatically scanning the modules to be loaded under a specified path and filtering out the subclasses of the base class that can be loaded includes: Recursively traverse the program files under the preset scan path, exclude program files and initialization files whose paths contain preset directories, keep program files ending with preset file types as files to be scanned, and generate a list of files to be scanned. By traversing the list of files to be scanned, each file is loaded and the target class corresponding to the file identifier is obtained; when the target class is a subclass of a loadable base class, the target class is added to the set of classes to be registered. The target classes belonging to the factory class in the set of classes to be registered are instantiated, and written into the dependency injection container with the factory class name or a custom name as the key to form operator factory instances corresponding to each operator.

8. The DAG operator extension method based on design patterns according to claim 1, characterized in that, The chain of responsibility model includes: By creating load method decorators, target load methods that need to be included in the configuration loading process are marked, and loading priorities are set for each target load method. During system initialization, the target loading methods in the loader class are scanned, and the corresponding loading methods and loading priorities are extracted to form a set of loading methods to be executed. The set of loading methods to be executed is sorted in ascending order according to loading priority, a chain of responsibility loader class is constructed, and the sorted target loading methods are written into the chain of responsibility loader class. Loading is performed sequentially according to the order of the target loading methods in the chain of responsibility loader class.

9. A design pattern-based DAG operator extension system, used to implement the design pattern-based DAG operator extension method according to any one of claims 1-8, characterized in that, The system includes: The design pattern layer is used to associate and register business logic classes with the original operator classes in the DAG workflow to be executed, using the decorator pattern to dynamically generate an operator class hierarchy including operator classes, proxy classes, and factory classes; it uses the factory pattern to determine the creation method of proxy instances corresponding to each original operator class, and uses the dependency injection pattern to scan, instantiate, and register the factory class with the container to obtain operator factory instances corresponding to each operator; it uses the proxy pattern to control the operator execution process and enhance aspects of the proxy class; and it uses the chain of responsibility pattern to load the operator configuration information required by each original operator class in an orderly manner. The DAG operator execution engine layer is used to parse the dependency relationships and execution order between each original operator class according to the DAG workflow to be executed, obtain the corresponding operator factory instance from the dependency injection container according to the execution order and generate the corresponding proxy instance, call the proxy instance to execute the corresponding operator, and write the execution result of the current operator into the input of the subsequent operator until all operators of the workflow are executed. The business application layer is used to summarize the execution results of each operator, obtain the target processing result, and return the target processing result as the output result of the DAG workflow to be executed.

10. An electronic device, characterized in that, The electronic device includes: processor; System bus; Main memory that is communicatively connected to the processor via the system bus; An auxiliary memory that is communicatively connected to the processor via the system bus; The main memory or the auxiliary memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor to implement the method of any one of claims 1-8.

Images