Compatible stream batch data processing large model workflow construction method

Abstract

The application relates to a large model workflow construction method compatible with stream batch data processing. The method is compatible with stream batch data processing through unified data block encapsulation, and unifies data output result forms. Decoupling is adopted between modules, and data governance and workflow construction and the like can be completely replaced by platforms with similar functions. Data governance and workflow construction can be processed through visualized dragging and dropping of operators, so that a use threshold is reduced, and iteration efficiency of the large model workflow is improved. In the method, the workflow can be periodically run in a self-defined mode, so that a daily manual maintenance problem is solved. Deployment complexity is low: complex environment configuration and maintenance are not required, and one-key deployment can be realized through docker-composition.

Description

Technical Field

[0001] This invention belongs to the field of rapid construction technology of data intelligent processing workflow involving large models, and relates to a method for constructing a large model workflow compatible with both streaming and batch data processing. Background Technology

[0002] With the rapid development of large model capabilities, various large model applications are becoming more and more common. Large model workflows, as an effective way for large models to participate in solving practical problems, have achieved a leap from "single model reasoning" to "intelligent pipelines," significantly reducing the threshold for large model applications and improving development iteration efficiency.

[0003] Reference 1 (Publication No.: CN119597271A) proposes a system for intelligent dialogue based on a large model-driven workflow. The system includes: a node configuration module, used to select one or more nodes and configure them in the configuration area according to the information displayed in the current user interface configuration layer, forming a workflow; a plugin import module, used to import data from the plugin nodes in the workflow to describe their input and output parameter formats; a context variable filling module, used to fill the input and output variables of nodes according to different types when nodes are executed in the workflow; and a workflow execution module, used to parse the nodes and their connections in the workflow and execute the processing logic of the nodes according to the specified workflow order. This system solves the problems of long delivery times, fixed use cases, and high development and maintenance thresholds in existing AI application development models.

[0004] Reference 2 (Publication No.: CN119991000A) proposes a method for implementing an internal audit AI assistant based on a large model and workflow. This method includes the following steps: setting up the AI ​​assistant information for AEO internal audits; this setting includes setting assistant prompts, an assistant model, an assistant knowledge base, and a preset workflow; configuring external data accounts required for auditing and adding them to the assistant knowledge base; uploading archived audit materials via dialogue, performing a completeness check using the preset workflow and the large model, and responding with audit comments; if incomplete, accessing external data through the configuration, automatically collecting data, and automatically archiving the data into the current audit data; its advantages are: compared to existing manual auditing methods, it greatly improves efficiency and reduces manpower and time costs; simultaneously, by configuring access to external data and automatically archiving it into the current audit data, it can more effectively solve problems.

[0005] Reference 3 (Publication No.: CN119692954A) proposes an intelligent workflow interaction configuration method based on a large model. This method includes: acquiring user function implementation requirements information and constructing a comprehensive large model by combining multiple pre-set models, knowledge bases, and plugins. The large model can output corresponding output requirements information based on input requirements information and supports knowledge retrieval and configuration. The construction process involves steps such as model selection, knowledge base filtering, plugin integration, and parameter adjustment. The output process includes the generation, evaluation, decision analysis, and optimization selection of workflow configuration schemes. This invention achieves intelligent and automated workflow configuration, improving configuration efficiency and accuracy.

[0006] Problems in the development of existing large-scale model applications: 1) Streaming and batch data require different technology stacks for processing, resulting in high development and maintenance costs; 2) Workflows require repeated development of operators, especially for processing different data types, leading to poor process reusability; 3) Workflow construction depends on code development, which cannot be participated in by non-technical personnel, resulting in high development costs; 4) Traditional system deployment environments and processes are complex and difficult to scale elastically. Summary of the Invention

[0007] To address the problems existing in the traditional methods mentioned above, this invention proposes a method for building large model workflows that are compatible with both streaming and batch data processing. Through data block encapsulation and drag-and-drop workflow rapid iterative design, it achieves compatible processing of streaming and batch data, low-code construction of large model workflows, and supports containerized one-click deployment and elastic scaling.

[0008] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a method for constructing a large model workflow compatible with both streaming and batch data processing is provided, including the following steps: Step S1: Construct a unified data block encapsulation method that is compatible with the processing of batch and streaming data, and unify the data output result format.

[0009] Step S2: Using a low-code visual interface, pre-built operators are freely arranged and encapsulated through drag-and-drop, constructing a large model workflow and ultimately providing a unified API interface to the outside world; Step S3: Enter the scheme name, select the data block to be processed by the scheme, select the workflow according to the selected data block, configure the scheme parameters, and confirm the parameters.

[0010] Step S4: Use Docker to containerize the data access layer, workflow engine, and front-end and back-end service modules, and deploy them with one click using docker-compose.

[0011] In one embodiment, step S1 includes: batch data comprising: data tables in Hive and MySQL databases, each data table containing four essential fields: data block content, data block time, data block type, and data block identifier; the data block content is organized text content to be processed by the large model workflow; the data block time is the time when the data was generated, in the format yyyy-MM-dd HH:mm:ss; the data block type is the data type used to distinguish the source of the data; the data block identifier is a unique identifier for the data; the streaming data is real-time streaming data consumed from Kafka, containing four essential fields: data block content, data block time, data block type, and data block identifier; the data block content, data block time, data block type, and data block identifier constitute the smallest unit of a database.

[0012] In one embodiment, step S2 includes: the platform provides an operator library, which includes a large number of pre-built operators, including: large model call operators, API call operators, Python script operators, conditional branch operators, loop operators, and adding comments; the platform uses a low-code visual interface to freely arrange the pre-built operators through drag-and-drop to form a DAG workflow that meets the task requirements; after the workflow is built, it is published to the outside as a unified API interface, encapsulating the internal processing logic and completing the decoupling of each module.

[0013] In one embodiment, step S3 includes: inputting a unique name for the scheme; Select the data block to be processed by the proposed solution; select the workflow required by the proposed solution to process the selected data block; configure parameters, including: the operation mode and number of threads of the solution; the operation mode of the solution includes one-time operation and periodic operation; the number of threads of the solution includes: single-threaded operation and multi-threaded operation; confirm the configured parameters and form a solution, and finally the production line will automatically start processing data according to the configuration.

[0014] In one embodiment, a one-time run refers to submitting all data in a data block to the workflow for processing in one batch; a periodic run refers to periodically retrieving data from a specified partition in the data block and submitting it to the workflow for processing.

[0015] In one embodiment, when multi-threaded execution is selected, multiple "workflows" are started for computation. In each run, the data in the "data block" is evenly distributed to each thread "workflow" for independent computation.

[0016] In one embodiment, the cluster is replaced by Kubernetes (k8s) to accommodate larger-scale cluster applications and maintenance.

[0017] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned method for constructing large-scale model workflows compatible with both streaming and batch data processing utilizes a unified data block encapsulation approach to ensure compatibility with both streaming and batch data processing, and standardizes the data output format. Modules are decoupled, allowing data governance and workflow construction modules to be replaced with similar functional platforms. Both data governance and workflow construction can be handled through visual drag-and-drop operators, lowering the barrier to entry and improving the iteration efficiency of large-scale model workflows. Workflows can be customized for periodic execution, eliminating the need for daily manual maintenance. Deployment complexity is low: no complex environment configuration or maintenance is required; deployment can be completed with a single click using docker-compose. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for constructing a large model workflow compatible with both batch and streaming data processing in one embodiment. Figure 2 This is a schematic diagram of the scheme creation process in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0022] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] The large-scale workflow construction method for compatible batch and streaming data processing proposed in this application runs on a platform that includes modules such as a data block construction module, a workflow building module, and a solution creation module. The data block construction module encapsulates and integrates open-source data components such as Hive and Kafka, supporting the processing of batch and streaming data. The workflow building module supports open-source process building components such as Dify and Coze. In the solution creation module, a solution requires specifying concrete data blocks and workflows. After a solution is created, the system automatically extracts data from the data blocks and calls the workflow for processing to generate the results.

[0025] In one embodiment, such as Figure 1 As shown, a method for constructing a large model workflow compatible with both streaming and batch data processing is provided, which may include the following processing steps S1 to S4: Step S1: Construct a unified data block encapsulation method that is compatible with the processing of streaming and batch data, and unify the data output result format.

[0026] Specifically, streaming data (such as Kafka) and batch data (such as Hive / MySQL) require different technology stacks (such as Flink and Spark), resulting in high development and maintenance costs and increased system complexity. Therefore, building a unified data block encapsulation method that is compatible with the processing of streaming and batch data and unifying the data output format can significantly reduce development and maintenance costs.

[0027] The data block building module encapsulates and integrates open-source data components such as Hive and Kafka, supporting the processing of batch and streaming data.

[0028] The data block building module can be replaced by any data modeling platform that conforms to the data block structure standard (i.e., the smallest unit that makes up a data block includes: data block identifier, data block time, data block content, and data block type).

[0029] Step S2: Using a low-code visual interface, pre-built operators are freely arranged and encapsulated through drag-and-drop, constructing a large model workflow and ultimately providing a unified API interface to the outside world.

[0030] Specifically, because existing technologies require the repeated development of operators for the same business logic across different data sources, workflow components cannot be reused across data types, especially when handling different data types, resulting in poor process reusability. Therefore, the workflow construction process specifically includes: Operator Library: The workflow building platform provides a large number of pre-built operators, such as large model operators (LLM calls), API call operators, Python script operators, etc.

[0031] Drag-and-drop arrangement: Operators can be freely dragged and dropped into a low-code visual interface to form a DAG workflow that meets various tasks.

[0032] Provide a unified API: Workflows can be published to form a unified API interface, making it convenient for external users to call them.

[0033] Large model workflows are built through drag-and-drop free arrangement, and a unified API interface is ultimately provided to the outside world, thereby achieving decoupling and improving processing efficiency.

[0034] This invention abstracts large-scale batch data processing into a modular design concept of "data blocks + workflow." The "data blocks" and "workflow" modules are decoupled from each other, and both modules can be replaced by platforms with similar functions, offering high flexibility and adaptability. "Data blocks" are streams / batch data organized and processed according to business needs, conforming to specific formats. "Workflow" is a data processing flow orchestrated using intelligent operators such as large models. "Data blocks" are the input data for the "workflow."

[0035] The construction of data blocks and workflows is entirely based on drag-and-drop free arrangement, which significantly shortens the construction time and meets efficiency requirements.

[0036] The workflow building block can also be replaced with any large-scale workflow orchestration platform that conforms to the API interface standard. The workflow module and workflow platform refer to the same thing; similar to Coze, a product of ByteDance.

[0037] Step S3: Enter the scheme name, select the data block to be processed, select the workflow according to the selected data block, configure the scheme parameters, and confirm the parameters.

[0038] Specifically, the process of creating a solution is as follows: enter the solution name, bind data blocks, bind workflow, configure the solution operation mode, configure the data details viewing address, and confirm parameters.

[0039] Automated data processing: Once the solution is developed, data processing runs automatically and periodically like an assembly line, reducing management and maintenance costs.

[0040] Each module can be replaced with a similar processing module, and the decoupling requirements are met. Modular design significantly reduces system maintenance costs and simplifies user operation.

[0041] Step S4: Use Docker to containerize the data access layer, workflow engine, and front-end and back-end service modules, and deploy them with one click using docker-compose.

[0042] Specifically, all modules of this invention have been containerized, and the docker-compose.yaml file has been written, which can be quickly deployed and started with one click using docker-compose, while also realizing persistent storage of data resources and elastic scaling of containers.

[0043] The entire system is containerized, supporting dynamic expansion and persistent storage, and offering stable performance.

[0044] The aforementioned method for constructing large-scale model workflows compatible with both streaming and batch data processing utilizes a unified data block encapsulation approach to ensure compatibility with both streaming and batch data processing, and standardizes the data output format. Modules are decoupled, allowing data governance and workflow construction modules to be replaced with similar functional platforms. Both data governance and workflow construction can be handled through visual drag-and-drop operators, lowering the barrier to entry and improving the iteration efficiency of large-scale model workflows. Workflows can be customized for periodic execution, eliminating the need for daily manual maintenance. Deployment complexity is low: no complex environment configuration or maintenance is required; deployment can be completed with a single click using docker-compose.

[0045] In one embodiment, step S1 includes: batch data comprising: data tables in Hive and MySQL databases, each data table containing four essential fields: data block content, data block time, data block type, and data block identifier; the data block content is organized text content to be processed by the large model workflow; the data block time is the time when the data was generated, in the format yyyy-MM-dd HH:mm:ss; the data block type is the data type used to distinguish the source of the data; the data block identifier is a unique identifier for the data; the streaming data is real-time streaming data consumed from Kafka, containing four essential fields: data block content, data block time, data block type, and data block identifier; the data block content, data block time, data block type, and data block identifier constitute the smallest unit of a database.

[0046] Specifically, the batch data format requirements are as follows: tables in common databases such as Hive and MySQL, containing four essential fields: data block content, data block time, data block type, and data block identifier. This table can come directly from various common databases or from data governance platforms that have already been processed, as long as it contains the four essential fields, facilitating subsequent module calls and processing.

[0047] Data streaming format requirements: This refers to Kafka data streams, which must be serialized JSON data in UTF-8 encoding and contain four necessary fields: data block content, data block time, data block type, and data block identifier.

[0048] In the above fields: Data block content is the organized text content to be processed by the large model workflow; Data block time is the time when the data was generated, in the format yyyy-MM-dd HH:mm:ss; Data block type is the type of data that can be used to distinguish the source of the data; Data block identifier, similar to UUID or rowkey, is an encoding that can uniquely identify the data.

[0049] These constitute the smallest unit of a data block.

[0050] In one embodiment, step S2 includes: the platform provides an operator library, which includes a large number of pre-built operators, including: large model call operators, API call operators, Python script operators, conditional branch operators, loop operators, and adding comments; the platform uses a low-code visual interface to freely arrange the pre-built operators through drag-and-drop to form a DAG workflow that meets the task requirements; after the workflow is built, it is published to the outside as a unified API interface, encapsulating the internal processing logic and completing the decoupling of each module.

[0051] Specifically, the workflow construction process includes: (1) Operator library: The platform provides a large number of pre-built operators, such as: 1) Large Model Call Operator (LLM): Encapsulates large model calls to lower the barrier to entry for using large models and facilitates rapid iteration of prompts when using them.

[0052] 2) API call operator: An operator that encapsulates various HTTP requests. When using it, simply fill in the URL and parameters.

[0053] 3) Python script operators: For functional operators not provided by the platform, you can use Python script operators to write simple Python scripts to implement them, ensuring flexibility and usability.

[0054] 4) Conditional branching operators: Based on the conditions you design, use if / elif / else to split the process into multiple branches.

[0055] 5) Loop operator: Executes a block of logic repeatedly until a condition is met or the maximum number of iterations is reached.

[0056] 6) Add notes: You can add notes in any location to make it easier for you to annotate and explain the processing logic, and also to better summarize and pass on experience.

[0057] (2) Free drag-and-drop arrangement: The platform adopts a low-code visual interface for the free drag-and-drop arrangement of operators, which lowers the threshold for users and allows people to focus on building their own processing logic without having to consider various details, ultimately forming a DAG workflow that meets various tasks.

[0058] (3) Provide a unified API: After the workflow of the platform is built, it can be published as a unified API interface to encapsulate the internal processing logic, decouple the modules, and facilitate external modules to call and process.

[0059] In one embodiment, such as Figure 2 As shown, step S3 includes: inputting a unique name for the solution; selecting the data block to be processed by the solution; selecting the workflow required by the solution to process the selected data block; configuring parameters, including: the solution's running mode and the number of threads; the solution's running mode includes one-time operation and periodic operation; the number of threads includes: single-threaded operation and multi-threaded operation; confirming the configured parameters and forming the solution, and finally the production line automatically starts processing data according to the configuration.

[0060] Specifically, a solution is an automated processing pipeline built based on a business scenario, in the form of "data blocks + workflow". The process of building a solution is as follows: ① Create a new scheme: Enter a unique name for the scheme.

[0061] ② Select data block: Select the data block to be processed in this scheme.

[0062] ③ Select Workflow: Select the workflow required for this scheme to process the data block.

[0063] ④ Configuration Parameters: The main parameters are the running mode and the number of threads. The solution can run in two ways: one-time execution and periodic execution. One-time execution submits all data in the "data block" to the "workflow" for processing in one batch. Periodic execution retrieves data from a specified partition from the "data block" periodically and submits it to the "workflow" for processing. The solution supports single-threaded and multi-threaded execution. If multi-threaded execution is selected, multiple "workflows" will be started for computation. In each run, the data in the "data block" will be evenly distributed to each thread "workflow" for independent computation.

[0064] ⑤ Confirm and generate the plan: Finally, confirm whether the parameters configured in the above steps are correct and generate the plan. Then the production line will automatically start processing data according to the configuration.

[0065] In one embodiment, a one-time run refers to submitting all data in a data block to the workflow for processing in one batch; a periodic run refers to periodically retrieving data from a specified partition in the data block and submitting it to the workflow for processing.

[0066] In one embodiment, when multi-threaded execution is selected, multiple "workflows" are started for computation. In each run, the data in the "data block" is evenly distributed to each thread "workflow" for independent computation.

[0067] In one embodiment, the cluster is replaced by Kubernetes (k8s) to accommodate larger-scale cluster applications and maintenance.

[0068] Specifically, the cluster can be replaced by Kubernetes (k8s) from Docker Swarm to accommodate larger-scale cluster applications and maintenance. The relevant steps are as follows: (1) Assessment and planning: Analyze and record the service topology, network configuration, storage volume, and resource usage in the existing Swarm cluster, determine the specific migration strategy to use parallel migration or gradual migration, and recommend retaining the Swarm cluster as a rollback strategy. (2) Build a K8s cluster, ensure that the node resources meet the requirements, and install necessary components such as K8s command line tools and Ingress Controller; (3) Convert the configuration file, convert the docker-compose.yml configuration file into K8s Deployment and Service resource files to realize K8s related service exposure, load balancing, storage mapping and security policy configuration; (4) Deploy application migration data, deploy relevant instances in K8s, back up the persistent storage data of the Swarm cluster and restore it in K8s, so as to ensure the continuity of related services and data; (5) Verification and monitoring: test and verify the functions of the service and platform, check whether the status is healthy and normal, monitor the performance after migration, and regularly use kubectl logs to check for any anomalies.

[0069] In one specific embodiment, a platform based on a large model performs compliance reviews on text and image content. This platform receives a large number of text and image posts daily, and the backend needs to conduct compliance reviews first. Traditional methods, such as keyword filtering and manual review, are prone to missing a large amount of content that cleverly bypasses the review process. The specific steps include: (1) Organize the data by organizing the data to be reviewed each day into streaming data blocks; (2) Construct a review model. Use the workflow construction module to build the model, which mainly includes data preprocessing, large model review, prompt word iteration, output content review conclusions, etc. (3) Scheme construction: combine data blocks and workflows to form a content review scheme. For those that are indeed problematic, they will be directly prohibited from being published. For those that are questionable, the submitters will be reviewed. (4) Continuously optimize and iterate the workflow to achieve the best desired results.

[0070] Other uses of this method: (1) Automated log auditing: It can easily filter out illegal operations and various behaviors that attempt to evade auditing. (2) Financial compliance: It can process bank transaction data in real time and call large models to identify abnormal behaviors such as money laundering. (3) Intelligent recommendation: It can be used to build recommendation systems and realize personalized recommendation services through custom workflows and large model prompts.

[0071] It should be understood that, although the above process Figure 1 The steps in the diagram are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above process... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A method for constructing a large-scale model workflow compatible with both streaming and batch data processing, characterized in that, Including the following steps: Step S1: Construct a unified data block encapsulation method that is compatible with the processing of streaming and batch data, and unify the data output result format; Step S2: Using a low-code visual interface, pre-built operators are freely arranged and encapsulated through drag-and-drop, constructing a large model workflow and ultimately providing a unified API interface to the outside world; Step S3: Enter the scheme name, select the data block to be processed, select the workflow according to the selected data block, configure the scheme parameters, and confirm the parameters; Step S4: Use Docker to containerize the data access layer, workflow engine, and front-end and back-end service modules, and deploy them with one click using docker-compose.

2. The method for constructing a large-scale model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, Step S1 includes: The batch data includes data tables in Hive and MySQL databases. Each data table contains four essential fields: data block content, data block time, data block type, and data block identifier. The data block content is the organized text content to be processed by the large model workflow. The data block time is the time the data was generated, in the format yyyy-MM-ddHH:mm:ss. The data block type is the data type used to distinguish the data source. The data block identifier is a unique code that identifies the data. Streaming data is real-time streaming data consumed from Kafka, and contains four essential fields: data block content, data block time, data block type, and data block identifier. The data block content, the data block time, the data block type, and the data block identifier constitute the smallest unit of a database.

3. The method for constructing a large model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, Step S2 includes: The platform provides an operator library, which includes a large number of pre-built operators, including: large model call operators, API call operators, Python script operators, conditional branch operators, loop operators, and the ability to add comments. The platform uses a low-code visual interface to create DAG workflows that meet task requirements by dragging and dropping pre-defined operators that can be freely arranged and encapsulated. Once the workflow is built, it is published as a unified API interface, encapsulating the internal processing logic and decoupling the various modules.

4. The method for constructing a large model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, Step S3 includes: Enter a unique name for the solution; Select the data block to be processed by the proposed scheme; The workflow required to process the selected data blocks is chosen using the aforementioned scheme. Configuration parameters, including: the scheme's running mode and number of threads; the running mode includes one-time running and periodic running; the number of threads includes: single-threaded running and multi-threaded running; Once the configured parameters are confirmed and a plan is formulated, the production line will automatically begin processing data according to the configuration.

5. The method for constructing a large model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, A one-time run refers to submitting all data in a data block to the workflow for processing in one batch; a periodic run refers to periodically retrieving data from a specified partition in the data block and submitting it to the workflow for processing.

6. The method for constructing a large-scale model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, When multi-threaded execution is selected, multiple "workflows" will be started for computation. In each run, the data in the "data block" will be evenly distributed to each thread "workflow" for independent computation.

7. The method for constructing a large model workflow compatible with batch and streaming data processing according to claim 1, characterized in that, Replace the cluster with Docker Swarm to adapt to larger-scale cluster applications and maintenance.

Images