Data closed-loop workflow processing method and electronic equipment

Abstract

The invention provides a data closed-loop workflow processing method and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that if a data closed-loop test task meets a preset test triggering condition, a preset workflow template for the data closed-loop test task is obtained, a to-be-analyzed data source of the data closed-loop test task is split, and a to-be-analyzed workflow template is obtained; obtaining a plurality of subtasks; according to a preset workflow template and the multiple subtasks, workflow parameters of the multiple subtasks are generated, the preset workflow template comprises work nodes of at least two jobs, the work nodes of the at least two jobs are execution nodes with the execution sequence meeting the preset data flow sequence, and according to the workflow parameters of the multiple subtasks, the work nodes of the at least two jobs are execution nodes with the execution sequence meeting the preset data flow sequence. And an application program interface API of the preset workflow execution engine is called, and the preset workflow execution engine is adopted to execute the plurality of subtasks in parallel according to the workflow parameters of the plurality of subtasks, so that the efficiency of data closed-loop workflow processing is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data closed-loop workflow processing method and electronic device. Background Art

[0002] As the core function of data closed-loop algorithms, data closed-loop scheduling requires multi-level processing and analysis of diverse data for a single task. These processes include, but are not limited to, data reception, cleaning, sharding, feature mining, re-injection, and model evaluation.

[0003] Currently, algorithms for executing data closed-loops lack the ability to decompose big data tasks and perform concurrent control, resulting in low server resource utilization. Furthermore, there is a lack of flexible configuration options for the diverse data processing flows of different users, requiring additional development to adapt, increasing maintenance costs and technical complexity. Summary of the Invention

[0004] The purpose of the present invention is to provide a data closed-loop workflow processing and electronic device to address the deficiencies in the above-mentioned prior art, so as to split the data source to be analyzed of the data closed-loop test task into multiple concurrently executed subtasks, thereby improving the efficiency of data closed-loop workflow processing.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides a data closed-loop workflow processing method, the method comprising: If the data closed-loop test task meets the preset test trigger condition, obtain a preset workflow template for the data closed-loop test task, and split the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; Generating workflow parameters for the multiple subtasks based on the preset workflow template and the multiple subtasks, wherein the preset workflow template includes: working nodes of at least two jobs, the working nodes of the at least two jobs are execution nodes whose execution order satisfies a preset data flow order, and the workflow parameters of each subtask include: task parameters of the working nodes of the at least two jobs; According to the workflow parameters of the multiple subtasks, an application program interface (API) of a preset workflow execution engine is called, and the preset workflow execution engine is used to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks.

[0006] In an optional embodiment, the preset workflow template further includes: data source configuration information; the data source to be analyzed of the data closed-loop test task is split to obtain multiple subtasks, including: Determining the size of the data source to be analyzed according to the data source configuration information; The data source to be analyzed is split according to the size of the data source to be analyzed to obtain the multiple subtasks.

[0007] In an optional embodiment, after splitting the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks, the method further includes: The task identifier of the data closed-loop test task is added to the execution tags of the multiple subtasks to mark the multiple subtasks after being split.

[0008] In an optional embodiment, the generating workflow parameters of the plurality of subtasks according to the preset workflow template and the plurality of subtasks, the method further includes: According to a preset concurrent job number, the plurality of subtasks are grouped to obtain at least one group of subtasks, wherein the same group of subtasks is a group of subtasks that are started and executed concurrently; According to the preset workflow template and each group of subtasks, workflow parameters of each subtask in each group of subtasks are generated.

[0009] In an optional embodiment, the number of work nodes for each job is at least two; generating workflow parameters for each subtask in each group of subtasks based on the preset workflow template and each group of subtasks includes: Obtain the number of concurrent nodes for each job, where the number of concurrent nodes for each job is less than or equal to the number of working nodes for the corresponding job; Determining, based on the preset workflow template, each group of subtasks, and the number of concurrent nodes of each job, a target working node for each subtask for each job from at least two working nodes of each job; Based on the information of the target working node of each subtask for each job and each group of subtasks, the workflow parameters of each subtask in each group of subtasks are generated, wherein the workflow parameters of each subtask also include: information of the target working node of the corresponding subtask for the at least two jobs.

[0010] In an optional embodiment, at least two execution algorithms for the same job are pre-configured on the working node of each job; Generating workflow parameters for the multiple subtasks according to the preset workflow template and the multiple subtasks includes: Obtaining node algorithm configuration parameters for each of the jobs; Determining a target execution algorithm for each job from at least two execution algorithms for each job according to the node algorithm configuration parameters of each job; The workflow parameters of the multiple subtasks are generated according to the preset workflow template, the multiple subtasks and the target execution algorithm information of each job, wherein the workflow parameters of each subtask also include: the target execution algorithm information of the at least two jobs.

[0011] In an optional embodiment, the using the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks includes: Using the preset workflow execution engine, calling a preset container platform according to the workflow parameters of each subtask, so that the preset container platform uses the container corresponding to each subtask, calls the execution algorithm of each subtask on the working node of the two jobs, and obtains the execution log of each subtask; After each subtask is completed, the preset workflow execution engine is used to perform persistence processing on the execution log of each subtask.

[0012] In an optional embodiment, the method further comprises: After each subtask is completed, the capacity resources of each subtask are recycled using the preset container platform according to a preset container recycling strategy.

[0013] In an optional embodiment, the method further comprises: Calling the API of the preset workflow execution engine according to the task identifiers of the multiple subtasks to obtain the execution status of the multiple subtasks; Calculating the execution progress of the multiple subtasks and the execution progress of the data closed-loop test task according to the execution status of the multiple subtasks; A graph editing engine is used to display the execution status of the multiple subtasks, the execution progress of the multiple subtasks, and the execution progress of the data closed-loop test task through a visual status display interface.

[0014] In an optional embodiment, if the data closed-loop test task meets the preset test trigger condition, before obtaining the preset workflow template for the data closed-loop test task, the method further includes: Using a graph editing engine to obtain an initial workflow template through the workflow configuration interface; Using the graph editing engine to configure at least two job work node components for the initial workflow template; Using the graph editing engine to configure execution configuration algorithms and execution configuration parameters for the work node components of the at least two jobs respectively; A target workflow template is generated according to the execution configuration algorithm and the execution configuration parameters of the work node components of the at least two jobs.

[0015] In a second aspect, an embodiment of the present application further provides a data closed-loop workflow processing device, comprising: A splitting module is used to obtain a preset workflow template for the data closed-loop test task if the data closed-loop test task meets the preset test trigger condition, and split the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; a generation module, configured to generate workflow parameters for the plurality of subtasks based on the preset workflow template and the plurality of subtasks, wherein the preset workflow template includes: work nodes of at least two jobs, the work nodes of the at least two jobs being execution nodes whose execution order satisfies a preset data flow order, and the workflow parameters of each subtask include: task parameters of the work nodes of the at least two jobs; The execution module is used to call the application program interface API of a preset workflow execution engine according to the workflow parameters of the multiple subtasks, and use the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks.

[0016] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the program instructions to perform the steps of the data closed-loop workflow processing method as described in any one of the first aspects.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data closed-loop workflow processing method as described in any one of the first aspects are executed.

[0018] The beneficial effects of this application are: The embodiment of the present application provides a data closed-loop workflow processing method and electronic device, the method comprising: if a data closed-loop test task meets a preset test trigger condition, obtaining a preset workflow template for the data closed-loop test task, and splitting the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; generating workflow parameters for multiple subtasks according to the preset workflow template and the multiple subtasks, wherein the preset workflow template includes: work nodes of at least two jobs, the work nodes of at least two jobs are execution nodes whose execution order meets the preset data flow order, and the workflow parameters of each subtask include: task parameters of the work nodes of at least two jobs; according to the workflow parameters of the multiple subtasks, calling the application program interface API of the preset workflow execution engine, and using the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks. The method of the present application greatly improves the efficiency of data closed-loop workflow processing by splitting the data source to be analyzed of the data closed-loop test task into multiple concurrently executed subtasks, adopts the containerization technology of the preset workflow execution engine to improve the reliability of the execution of multiple subtasks, and adopts the preset workflow template to realize targeted workflow processing of the data closed-loop test task. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 One of the flow diagrams of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 2 The second flowchart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 3 The third flowchart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 4 A fourth flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 5 A fifth flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 6 Flowchart 6 of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 7Flowchart 7 of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 8 8. Flowchart diagram of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 9 A schematic diagram of the functional modules of a data closed-loop workflow processing device provided in an embodiment of the present application; Figure 10 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0022] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0023] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the application is usually placed when in use. It is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on this application.

[0024] In addition, the terms "first," "second," and the like in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0025] It should be noted that, in the absence of conflict, the features in the embodiments of this application can be combined with each other.

[0026] The data closed-loop workflow processing method provided by the embodiment of the present application is explained in detail below with reference to the accompanying drawings through specific examples. Figure 1 This is one of the flow diagrams of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 1 As shown, the method includes: S101: If the data closed-loop test task meets the preset test trigger condition, obtain a preset workflow template for the data closed-loop test task, and split the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks.

[0027] In this embodiment, the data closed-loop test task may be a data closed-loop test task generated by different businesses. For example, if the data closed-loop test task is a data closed-loop test task generated by an autonomous driving test, the data source to be analyzed of the data closed-loop test task stores vehicle test data corresponding to multiple time periods. Each vehicle test data includes: vehicle driving route, lane data, etc. The data closed-loop test task specifically cleans, slices, mines, re-injects, evaluates, etc. each vehicle test data to achieve closed-loop processing of each vehicle test data.

[0028] The preset test trigger conditions are divided into scheduled test trigger conditions and manual test trigger conditions. If the preset test trigger condition is a scheduled test trigger condition, the data closed-loop test task is detected regularly. When a data closed-loop test task is detected, the preset workflow template for the data closed-loop test task is obtained and the data closed-loop test task is processed. For example, at 3:00 a.m. every day, it automatically checks whether there are new data closed-loop test tasks. If the preset test trigger condition is a manual test trigger condition, the manually uploaded data closed-loop test task is detected, the preset workflow template for the data closed-loop test task is obtained, and the data closed-loop test task is processed.

[0029] Among them, the preset workflow template is the blueprint for data closed-loop processing, which predefines the standard process for processing data closed-loop test tasks. Each data closed-loop test task has a corresponding workflow template, so the preset workflow template corresponding to the data closed-loop test task is obtained.

[0030] Since there are multiple data in the data source to be analyzed, multiple subtasks are obtained by splitting the data source to be analyzed. Each subtask includes at least one data. By splitting the data source to be analyzed, parallel processing of multiple subtasks can be achieved.

[0031] S102: Generate workflow parameters for multiple subtasks according to a preset workflow template and multiple subtasks.

[0032] Among them, the preset workflow template includes: working nodes of at least two jobs, the working nodes of at least two jobs are execution nodes whose execution order meets the preset data flow order, and the workflow parameters of each subtask include: task parameters of the working nodes of at least two jobs.

[0033] Specifically, the work nodes in the preset workflow template are the specific execution units of the data processing flow. Each work node represents a job, such as data cleaning, data slicing, or data mining. These jobs have a strict execution order, which is the preset data flow order. Taking data cleaning and data slicing as an example, data cleaning must be completed first to remove duplicate and erroneous data before further data slicing processing can be performed.

[0034] Based on the preset workflow template and the subtasks obtained by splitting, unique workflow parameters are generated for each subtask. For each subtask, its workflow parameters will describe in detail the specific operations and required data to be performed at each work node of the preset workflow template. For example, in a subtask, at the data cleaning work node, the workflow parameters will specify the cleaning rules (such as removing data records with empty fields), the input data source (that is, the original data corresponding to the subtask), and the output data storage location; at the data slicing work node, the slicing algorithm (such as calculating the data mean and standard deviation), the input data (that is, the cleaned data), and the output format of the slicing results will be clearly defined.

[0035] S103: Calling an API of a preset workflow execution engine according to workflow parameters of the multiple subtasks, and using the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks.

[0036] The default workflow execution engine is Argo Workflow, an open-source container-native workflow engine. Argo Workflow runs as a custom resource definition (Kubernetes, K8s), leveraging K8s' scheduling, resource management, and elastic scaling capabilities. Each workflow step is executed as a container, supporting containerized applications.

[0037] The application programming interface (API) of the pre-configured workflow execution engine serves as a bridge for interaction between the workflow execution system and the pre-configured workflow execution engine. The workflow execution system encapsulates the generated workflow parameters for multiple subtasks, known as ArgoWorkflow parameters, in the format specified by the API and then sends a call request to the pre-configured workflow execution engine. Specifically, the argo-client-java Github project is used as the SDK. The ApiClient is configured to specify the deployment location of the Argo Workflow environment. The API is called workflowServiceCreateWorkflow to create and execute workflows.

[0038] After receiving the workflow parameters for each subtask, the preset workflow execution engine invokes the capabilities of the Kubernetes cluster based on its own scheduling algorithm and resource allocation strategy to execute multiple subtasks in parallel. Parallel execution fully utilizes the system's computing resources, significantly reducing the processing time for data closed-loop testing tasks. During execution, the workflow execution engine strictly follows the order of the work nodes defined in each subtask's workflow parameters, calling the corresponding job execution programs one by one. For example, the data cleansing job executes first, passing the cleaned data to the data slicing job of the next work node for processing.

[0039] In summary, an embodiment of the present application provides a data closed-loop workflow processing method, which includes: if the data closed-loop test task meets the preset test trigger condition, obtaining a preset workflow template for the data closed-loop test task, and splitting the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; generating workflow parameters for multiple subtasks according to the preset workflow template and the multiple subtasks, wherein the preset workflow template includes: working nodes of at least two jobs, the working nodes of at least two jobs are execution nodes whose execution order meets the preset data flow order, and the workflow parameters of each subtask include: task parameters of the working nodes of at least two jobs; according to the workflow parameters of the multiple subtasks, calling the application program interface API of the preset workflow execution engine, and using the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks. The method of the present application greatly improves the efficiency of data closed-loop workflow processing by splitting the data source to be analyzed of the data closed-loop test task into multiple concurrently executed subtasks, adopts the containerization technology of the preset workflow execution engine, improves the reliability of the execution of multiple subtasks, and adopts the preset workflow template to realize targeted workflow processing of the data closed-loop test task.

[0040] This application also provides another possible implementation of a data closed-loop workflow processing method. The preset workflow template also includes: data source configuration information, Figure 2 This is a flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 2 As shown, the data source to be analyzed for the data closed-loop test task is split into multiple subtasks, including: S201: Determine the size of the data source to be analyzed according to data source configuration information.

[0041] S202: Split the data source to be analyzed according to the size of the data source to be analyzed to obtain multiple subtasks.

[0042] In this embodiment, the preset workflow template is pre-configured with data source configuration information, the size of the data source to be analyzed is determined according to the data source configuration information, and then the data source to be analyzed is split to obtain a certain number of subtasks of the same size.

[0043] It should be noted that the amount of data in the data source to be analyzed is determined according to the data source configuration information. For example, if the data source to be analyzed includes 100 data, the data source to be analyzed is split into 20 subtasks, each of which includes 5 data.

[0044] Optionally, after splitting the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks, the method further includes: The task identifier of the data closed-loop test task is added to the execution labels of multiple subtasks to mark the multiple subtasks after splitting.

[0045] Specifically, the metadata.lables property in Argo Workflow is used to add the task identifier of the data closed-loop test task to the execution labels of multiple subtasks to mark the multiple subtasks after splitting.

[0046] In the method provided in the embodiment of the present application, the size of the data source to be analyzed is determined based on the data source configuration information, and the data source to be analyzed is split according to the size of the data source to be analyzed to obtain multiple subtasks. By splitting the data source to be analyzed, multiple subtasks can be processed in parallel, thereby improving execution efficiency.

[0047] This application also provides another possible implementation of a data closed-loop workflow processing method. Figure 3 The third flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 3 As shown, according to the preset workflow template and the multiple subtasks, workflow parameters of the multiple subtasks are generated, and the method further includes: S301. Group multiple subtasks according to a preset concurrent job number to obtain at least one group of subtasks, where the same group of subtasks is a group of subtasks that are started and executed concurrently.

[0048] S302: Generate workflow parameters for each subtask in each group of subtasks based on the preset workflow template and each group of subtasks.

[0049] In this embodiment, a preset job concurrency is preconfigured, and multiple subtasks are grouped to obtain at least one group of subtasks, which are then started and executed concurrently. For example, if 10 subtasks are split and the preset job concurrency is 3, then the 10 subtasks are grouped to obtain at least one group of subtasks. The first group of subtasks includes 3 subtasks. In this case, the three subtasks in the first group can be started and executed concurrently. Once the three subtasks are completed simultaneously, the three subtasks in the second group continue to be started and executed concurrently until all subtasks are completed.

[0050] It should be noted that if one of the three subtasks in the first group of subtasks has been completed and the other two subtasks have not yet been completed, then one subtask in the second group of subtasks can start execution, and the other two subtasks need to wait for the other two subtasks in the first group of subtasks to be completed.

[0051] Then, based on the preset workflow template and each group of subtasks, the workflow parameters of each subtask in each group are generated, and the concurrent execution order of each subtask is clarified. This grouping mechanism ensures that resources are efficiently utilized while avoiding resource competition and performance degradation caused by excessive concurrency. It is suitable for processing large-scale data closed-loop testing tasks.

[0052] The present application also provides another possible implementation of a data closed-loop workflow processing method, where the number of working nodes for each job is at least two. Figure 4 This is a flowchart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 4 As shown, based on the preset workflow template and each group of subtasks, the workflow parameters of each subtask in each group of subtasks are generated, including: S401. Obtain the concurrent node count of each job. The concurrent node count of each job is less than or equal to the number of working nodes of the corresponding job.

[0053] In this embodiment, the node concurrency for each job is preconfigured in the preset workflow template, which can improve concurrency while ensuring resource security. Node concurrency refers to the number of worker nodes allowed to execute simultaneously in the same job. For example, in a data cleaning job, assuming there are 5 worker nodes, if the node concurrency is set to 3, a maximum of 3 worker nodes can execute simultaneously.

[0054] The node concurrency of each job is less than or equal to the number of working nodes of the corresponding job, so that at least one working node is executed and resources are not wasted due to the concurrency exceeding the number of nodes.

[0055] S402: Determine a target working node for each subtask for each job from at least two working nodes of each job according to a preset workflow template, each group of subtasks, and the number of concurrent nodes of each job.

[0056] Specifically, the target worker node is determined based on the number of concurrent nodes. The node that needs to execute the current subtask is selected from the multiple worker nodes of each job. Factors such as data dependencies, node priorities, and system resource availability are comprehensively considered to ensure the correct and efficient execution of tasks.

[0057] S403: Generate workflow parameters for each subtask in each group of subtasks based on the target working node information of each subtask for each job and each group of subtasks.

[0058] The workflow parameters of each subtask also include information about target working nodes of at least two jobs corresponding to the subtask.

[0059] Specifically, the workflow parameters of each subtask contain detailed information required for the execution of each subtask, among which the information of the target working nodes of the corresponding subtask for at least two jobs is the key part. Then, the relevant information of the target working nodes determined by each subtask for each job, such as node name, input and output data requirements, execution script or program, etc., are integrated into the workflow parameters.

[0060] In the method provided in the embodiment of the present application, by setting the node concurrency number for each job, the target working node of each subtask for each job can be determined, the workflow parameters of each subtask in each group of subtasks can be generated, and resources can be reasonably allocated to batch tasks.

[0061] The present application also provides another possible implementation of a data closed-loop workflow processing method, in which at least two execution algorithms of the same job are pre-configured on the working node of each job. Figure 5 This is a flowchart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 5 As shown, based on the preset workflow template and multiple subtasks, workflow parameters for multiple subtasks are generated, including: S501: Obtain node algorithm configuration parameters for each job.

[0062] In this embodiment, the working node of each job is pre-configured with multiple execution algorithms, which can be dynamically switched according to the characteristics of the task, and each execution algorithm is configured with a corresponding execution server.

[0063] S502: Determine a target execution algorithm for each job from at least two execution algorithms for each job according to the node algorithm configuration parameters of each job.

[0064] S503: Generate workflow parameters for the multiple subtasks according to the preset workflow template, the multiple subtasks, and information about the target execution algorithm of each job.

[0065] The workflow parameters of each subtask also include information about target execution algorithms of at least two jobs.

[0066] Specifically, according to the node algorithm configuration parameters of each job, the target execution algorithm of each job is determined from at least two execution algorithms of each job, and then the information of the target execution algorithm of each job is added to the workflow parameters of multiple subtasks, so that the preset workflow execution engine calls the target execution algorithm of each job according to the information of the target execution algorithm of each job, thereby selecting different servers for different target execution algorithms to improve the algorithm running speed.

[0067] This application also provides another possible implementation of a data closed-loop workflow processing method. Figure 6 The sixth flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 6 As shown, a preset workflow execution engine is used to execute multiple subtasks in parallel according to the workflow parameters of the multiple subtasks, including: S601. Use the preset workflow execution engine to call the preset container platform according to the workflow parameters of each subtask, so that the preset container platform uses the container corresponding to each subtask, calls the execution algorithm of each subtask on the working node for the two jobs and obtains the execution log of each subtask.

[0068] In this embodiment, the preset workflow execution engine interacts with the preset container platform based on the workflow parameters of each subtask when executing the subtask. The preset container platform can be Kubernetes, which can provide an efficient and isolated operating environment to ensure the stable execution of the subtask.

[0069] The workflow execution engine extracts the required information from the subtask's workflow parameters, including the subtask's corresponding container image name and version number, as well as the resource configuration and environment variables required for container operation. Taking Kubernetes as an example, the execution engine generates the corresponding container Pod resource description file, which defines the container's parameters in detail. The generated Pod resource description file is submitted to the Kubernetes API server, which uses the scheduler to schedule the Pod to the appropriate node to run, thereby starting the container corresponding to the subtask.

[0070] Each subtask has a preconfigured execution algorithm for the two job's worker nodes. Once the container starts, it invokes the execution algorithm on the target worker node specified in the workflow parameters. For example, if the subtask involves a data cleaning job, the container will invoke the data cleaning algorithm on the worker node of the data cleaning job. The container invokes these algorithms by executing pre-written scripts or programs. The scripts pass in parameters such as the input and output data paths specified in the workflow parameters to ensure that the algorithms process the data as expected.

[0071] During subtask execution, the container records execution logs in real time, including the algorithm start time, key operations during execution, and any errors encountered. The pre-configured container platform provides log collection and management capabilities. The workflow execution engine can retrieve each subtask's execution log from the container platform through pre-configured interfaces or configurations. This container log information is then centrally stored for easy access and processing by the workflow execution engine.

[0072] S602: After each subtask is completed, a preset workflow execution engine is used to perform persistence processing on the execution log of each subtask.

[0073] Specifically, in the Argo Workflow configuration file, add the configuration of the high-performance distributed object storage system MinIO. After each subtask is completed, the preset workflow execution engine is used to persist the execution log of each subtask to ensure that the operation log is not lost and the computing server resources are not occupied.

[0074] Optionally, after each subtask is completed, the capacity resources of each subtask are recycled using a preset container platform according to a preset container recycling strategy.

[0075] Specifically, considering storage resource limitations, the workflow execution engine will develop log retention and cleanup strategies. After each subtask is completed, the execution engine will automatically clean up capacity resources and release storage resources.

[0076] In the method provided in the embodiment of the present application, the preset workflow execution engine realizes the parallel execution of subtasks based on workflow parameters and effectively manages the execution logs, thereby ensuring the efficient, stable operation and traceability of the data closed-loop workflow.

[0077] This application also provides another possible implementation of a data closed-loop workflow processing method. Figure 7 The seventh flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 7 As shown, the method further includes: S701: According to the task identifiers of the multiple subtasks, call the API of the preset workflow execution engine to obtain the execution status of the multiple subtasks.

[0078] In this embodiment, each subtask is assigned a unique task identifier when it is split, and the API of the preset workflow execution engine, namely workflowServiceListWorkflows, is called to obtain an execution list of the multiple subtasks after the split.

[0079] Specifically, enter the listOptionsLabelSelector parameter, where label = the task identifier. The query results are specified using the fields parameter, and an object of the IoArgoprojWorkflowV1alpha1WorkflowList type is returned. The execution status of multiple subtasks includes different stages of execution, including completed, in progress, not started, and failed.

[0080] S702: Calculate the execution progress of the multiple subtasks and the execution progress of the data closed-loop test task according to the execution status of the multiple subtasks.

[0081] Specifically, weights are pre-set for different nodes based on execution time. The execution progress of each subtask is equal to the number of completed nodes multiplied by the weight. The execution progress of the data closed-loop test task is then calculated based on the execution progress of each subtask.

[0082] Among them, if a subtask contains 3 work nodes, and each work node is assigned a different weight, representing its importance in the subtask or the proportion of the expected execution time. For example, the weight of the first node is 0.3, the weight of the second node is 0.5, and the weight of the third node is 0.2. When the execution status of the subtask is obtained, it is known that the first node has been completed, the second node is being executed, and the third node has not yet started, the execution progress of the subtask is calculated as follows: the sum of the weights of the completed work nodes, which is 0.3, so the current execution progress of the subtask is 30%. This calculation method based on work nodes and weights can more accurately reflect the actual execution progress of the subtask.

[0083] The execution progress of a data closed-loop test task depends on the execution progress of all its subtasks. For example, a data closed-loop test task contains five subtasks. Subtask 1 processes 20% of the total data volume, subtask 2 processes 30% of the total data volume, and so on. After obtaining the execution progress of each subtask, add up the execution progress of the subtasks to obtain the execution progress of the data closed-loop test task. Assuming that the execution progress of subtask 1 is 20%, the execution progress of subtask 2 is 30%, and the execution progress of other subtasks is 0, the execution progress of the data closed-loop test task is 50%.

[0084] S703: Use a graph editing engine to display the execution status of multiple subtasks, the execution progress of multiple subtasks, and the execution progress of the data closed-loop test task through a visual status display interface.

[0085] Specifically, the graph editing engine, AntV / X6, is the core tool for visual display. It provides functions for creating, editing, and rendering graphical interfaces. It converts data into intuitive graphs and charts. When displaying the execution status and progress of subtasks and tasks, the graph editing engine first constructs a visualization model based on the data structure, defining the mapping between graphical elements (such as nodes, links, and charts) and data.

[0086] Specifically, the query results and the calculation results of the task execution progress are converted into the AntV / X6 flowchart format and returned. For example, the overall status of the workflow is displayed at the top of the interface, the subtask table is displayed on the left, and the workflow execution status is displayed on the right. You can click on the subtask to display the execution progress of each node of the subtask, or display the execution status of each node.

[0087] It should be noted that you can view the algorithm execution log by opening a subtask node. You can view the logs of successfully executed nodes through the / artifact-files / {namespace} / archived-workflows / {uid} / {name} / outputs / main-logs interface, and the logs of executing nodes can be viewed by viewing / api / v1 / workflows / {namespace} / {name} / log.

[0088] The method provided in the embodiment of the present application realizes real-time monitoring of the execution status of subtasks, accurate calculation of the execution progress and visual display, helping users to intuitively and comprehensively grasp the execution process and progress of the data closed-loop test task.

[0089] This application also provides another possible implementation of a data closed-loop workflow processing method. Figure 8 FIG8 is a flow chart of a data closed-loop workflow processing method provided in an embodiment of the present application; Figure 8 As shown, if the data closed-loop test task meets the preset test trigger condition, before obtaining the preset workflow template for the data closed-loop test task, the method further includes: S801: Using a graph editing engine to obtain an initial workflow template through a workflow configuration interface.

[0090] In this embodiment, the workflow configuration interface is a visual platform for workflow template creation and management interactions, and it is designed and developed based on the functions provided by the graph editing engine. Users enter the interface through a browser or a specific client application. The interface usually has elements such as a template library browsing area, a search box, and template import / export buttons. Users can browse existing initial workflow templates in the template library. These initial templates may be built-in general templates, such as data cleaning templates, data analysis templates, or templates shared by other users or historically saved. Users can quickly filter related templates by entering keywords in the search box, or directly import customized initial template files (common formats such as JSON and YAML) from local files. When the user selects an initial workflow template and clicks to confirm the operation, the graph editing engine will load the template into the workspace for the user to perform subsequent editing and configuration.

[0091] S802: Use a graph editing engine to configure at least two job work node components for the initial workflow template.

[0092] Specifically, the graph editing engine provides a rich library of work node components. For example, the left side of the workflow creation page contains business components, while the right side primarily features a flowchart drawing board. You configure workflow templates by dragging pre-configured platform business components onto the drawing board. This requires a data source to be present, and execution can be scheduled or manual.

[0093] Business components include various types of job nodes, such as data acquisition nodes, data conversion nodes, data storage nodes, and data analysis algorithm nodes. Based on the actual needs of the data closed-loop testing task, users select appropriate node components from the node component library and add them to the initial workflow template. Users can drag and drop node components from the library into the template flow in the workspace, or select "Add Node" from the right-click menu and choose a specific component from the list. When adding node components, the graph editing engine automatically assigns a unique identifier to each node and records information such as the node type and location.

[0094] After adding the working node components, users need to determine the execution order and data flow between nodes, and connect the nodes using the wiring function provided by the graph editing engine. Users simply click the output port of a node, drag the mouse to the input port of another node, and release the mouse to complete the wiring operation. The graph editing engine automatically constructs a directed acyclic graph structure between the working nodes based on the wiring, clarifying the execution order of the nodes. For example, the output of the "cleaning node" is connected to the input of the "slicing node", indicating that data is collected first and then sliced; the output of the "slicing node" is connected to the input of the "mining node", ensuring that the sliced ​​data serves as the input of the mining node. Through this visual connection operation, users can easily design a data processing flow that conforms to the processing logic of the data closed-loop test task.

[0095] S803: Use a graph editing engine to configure execution configuration algorithms and execution configuration parameters for the working node components of at least two jobs respectively.

[0096] Each worker node component is associated with a variety of configurable execution algorithms to meet different business processing requirements. The graph editing engine provides users with an algorithm selection interface. Clicking on a worker node component displays a list of supported execution algorithms in the pop-up property settings window. Users upload algorithm files based on data characteristics and processing goals, or select a file path in a Git repository and the server corresponding to the algorithm. After selecting an algorithm, the graph editing engine records the algorithm information used by the node and displays the corresponding parameter configuration area based on the algorithm type.

[0097] After selecting the execution configuration algorithm, the user needs to configure specific parameters for the algorithm, such as the slicing algorithm can configure the slicing time. Different algorithms correspond to different parameter settings. The graph editing engine will display the corresponding parameter input box, drop-down menu, slider and other interactive elements in the property setting window according to the selected algorithm. Users enter or select appropriate parameter values ​​through these interactive elements. The graph editing engine will verify the validity of the parameters in real time, such as checking whether the syntax of the regular expression is correct and whether the parameter value is within a reasonable range. If the parameter setting is incorrect, the engine will give a prompt message to guide the user to make corrections to ensure that the algorithm can be executed correctly.

[0098] S804: Generate a target workflow template according to the execution configuration algorithm and execution configuration parameters of the work node components of at least two jobs.

[0099] Specifically, after users complete the execution configuration algorithms and parameter settings for all working node components, the graph editing engine collects detailed information about each node, including node identification, node type, connection relationships, execution configuration algorithm name, execution configuration parameters, etc. This information is then consolidated and packaged in a specific data format, typically generating template data in JSON or YAML format.

[0100] The integrated and encapsulated template data becomes the target workflow template, which the diagram editing engine stores in the database. Users can name the template and add descriptive information to facilitate subsequent search and management. When the data closed-loop test task meets the preset test trigger conditions, the target workflow template can be retrieved from the data as the basis for executing the data closed-loop test task. In addition, users can also share the target workflow template, export it for backup, and perform other operations, improving the template's reusability and portability, providing reference and convenience for other similar data closed-loop test tasks.

[0101] It should be noted that the data flow of the data closed-loop platform must meet a certain order, and the execution process of the workflow in the template must meet certain dependencies. By parsing the template configuration, abnormal processes are prompted, and the creation of the workflow template is rejected. After passing the inspection, it is saved to the database.

[0102] In the method provided in the embodiment of the present application, the entire process from initial template acquisition to target workflow template generation is completed with the help of a graph editing engine, providing accurate and personalized workflow guidance for the smooth execution of data closed-loop testing tasks.

[0103] The following continues to explain the data closed-loop workflow processing device and electronic device provided by any of the above embodiments of the present application. The specific implementation process and the technical effects produced are the same as those of the corresponding method embodiments mentioned above. For the sake of brief description, for the parts not mentioned in this embodiment, please refer to the corresponding content in the method embodiment.

[0104] Figure 9 This is a functional module diagram of a data closed-loop workflow processing device provided in an embodiment of the present application. Figure 9 As shown, the data closed-loop workflow processing device 100 includes: The splitting module 110 is used to obtain a preset workflow template for the data closed-loop test task if the data closed-loop test task meets the preset test trigger condition, and split the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; A generation module 120 is configured to generate workflow parameters for a plurality of subtasks based on a preset workflow template and a plurality of subtasks, wherein the preset workflow template includes: work nodes of at least two jobs, the work nodes of the at least two jobs being execution nodes whose execution order satisfies a preset data flow order, and the workflow parameters of each subtask include: task parameters of the work nodes of the at least two jobs; The execution module 130 is configured to call an application programming interface (API) of a preset workflow execution engine according to workflow parameters of the multiple subtasks, and use the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks.

[0105] Optionally, the preset workflow template also includes: data source configuration information; a splitting module 110, further used to determine the size of the data source to be analyzed based on the data source configuration information; and split the data source to be analyzed based on the size of the data source to be analyzed to obtain multiple subtasks.

[0106] Optionally, the device further comprises: Add a module to add the task identifier of the data closed-loop test task to the execution tags of multiple subtasks to mark the multiple subtasks after splitting.

[0107] Optionally, the generation module 120 is also used to group multiple subtasks according to the preset concurrent number of jobs to obtain at least one group of subtasks, and the same group of subtasks is a group of subtasks that are started and executed concurrently; based on the preset workflow template and each group of subtasks, the workflow parameters of each subtask in each group of subtasks are generated.

[0108] Optionally, the number of working nodes for each job is at least two; the generation module 120 is also used to obtain the node concurrency number of each job, and the node concurrency number of each job is less than or equal to the number of working nodes of the corresponding job; according to the preset workflow template, each group of subtasks and the node concurrency number of each job, the target working node of each subtask for each job is determined from at least two working nodes of each job; according to the information of the target working node of each subtask for each job, and each group of subtasks, the workflow parameters of each subtask in each group of subtasks are generated, wherein the workflow parameters of each subtask also include: information of the target working node of the corresponding subtask for at least two jobs.

[0109] Optionally, at least two execution algorithms for the same job are pre-configured on the working node of each job; the generation module 120 is also used to obtain the node algorithm configuration parameters of each job; based on the node algorithm configuration parameters of each job, the target execution algorithm of each job is determined from the at least two execution algorithms of each job; based on the preset workflow template, multiple subtasks and the information of the target execution algorithm of each job, the workflow parameters of multiple subtasks are generated, wherein the workflow parameters of each subtask also include: information of the target execution algorithms of at least two jobs.

[0110] Optionally, the execution module 130 is also used to use a preset workflow execution engine to call a preset container platform according to the workflow parameters of each subtask, so that the preset container platform uses the container corresponding to each subtask, calls the execution algorithm of each subtask on the working node for the two jobs and obtains the execution log of each subtask; after each subtask is executed, the preset workflow execution engine is used to persist the execution log of each subtask.

[0111] Optionally, the execution module 130 is further configured to reclaim the capacity resources of each subtask by using a preset container platform according to a preset container recycling strategy after each subtask is completed.

[0112] Optionally, the device further comprises: The acquisition module is used to call the API of the preset workflow execution engine according to the task identifiers of the multiple subtasks to obtain the execution status of the multiple subtasks; A calculation module is used to calculate the execution progress of multiple subtasks and the execution progress of the data closed-loop test task according to the execution status of multiple subtasks; The display module is used to display the execution status of multiple subtasks, the execution progress of multiple subtasks, and the execution progress of the data closed-loop test task through a visual status display interface using a graph editing engine.

[0113] Optionally, the device further comprises: An acquisition module is used to acquire an initial workflow template through a workflow configuration interface using a graph editing engine; A configuration module is configured to configure the work node components of at least two jobs for the initial workflow template using a graph editing engine; and to configure the execution configuration algorithm and execution configuration parameters for the work node components of the at least two jobs respectively using the graph editing engine; The generating module 120 is further configured to generate a target workflow template according to the execution configuration algorithms and execution configuration parameters of the working node components of at least two jobs.

[0114] The above-mentioned device is used to execute the method provided in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0115] The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0116] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present application, which can be used for data closed-loop workflow processing. Figure 10 As shown, the electronic device includes: a processor 210 , a storage medium 220 , and a bus 230 .

[0117] Storage medium 220 stores machine-readable instructions executable by processor 210. When the electronic device is running, processor 210 communicates with storage medium 220 via bus 230, and processor 210 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation methods and technical effects are similar and will not be repeated here.

[0118] Optionally, the present application further provides a storage medium 220 on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method embodiment are executed. The specific implementation and technical effects are similar and will not be repeated here.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, the functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional units.

[0122] The aforementioned integrated unit implemented as a software functional unit can be stored in a computer-readable storage medium. The software functional unit, stored in a storage medium, includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) or a processor to execute portions of the method steps described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a removable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data closed-loop workflow processing method, characterized in that: The method comprises: If the data closed-loop test task meets the preset test trigger condition, obtain a preset workflow template for the data closed-loop test task, and split the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks; Generating workflow parameters for the multiple subtasks based on the preset workflow template and the multiple subtasks, wherein the preset workflow template includes: working nodes of at least two jobs, the working nodes of the at least two jobs are execution nodes whose execution order satisfies a preset data flow order, and the workflow parameters of each subtask include: task parameters of the working nodes of the at least two jobs; According to the workflow parameters of the multiple subtasks, an application program interface (API) of a preset workflow execution engine is called, and the preset workflow execution engine is used to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks.

2. The method according to claim 1, characterized in that The preset workflow template also includes: data source configuration information; the data source to be analyzed of the data closed-loop test task is split to obtain multiple subtasks, including: Determining the size of the data source to be analyzed according to the data source configuration information; The data source to be analyzed is split according to the size of the data source to be analyzed to obtain the multiple subtasks.

3. The method according to claim 1, characterized in that After splitting the data source to be analyzed of the data closed-loop test task to obtain multiple subtasks, the method further includes: The task identifier of the data closed-loop test task is added to the execution tags of the multiple subtasks to mark the multiple subtasks after being split.

4. The method according to claim 1, wherein Generating workflow parameters for the multiple subtasks according to the preset workflow template and the multiple subtasks, the method further includes: According to a preset concurrent job number, the plurality of subtasks are grouped to obtain at least one group of subtasks, wherein the same group of subtasks is a group of subtasks that are started and executed concurrently; According to the preset workflow template and each group of subtasks, workflow parameters of each subtask in each group of subtasks are generated.

5. The method according to claim 4, characterized in that The number of working nodes of each job is at least two; generating workflow parameters of each subtask in each group of subtasks according to the preset workflow template and each group of subtasks includes: Obtain the number of concurrent nodes for each job, where the number of concurrent nodes for each job is less than or equal to the number of working nodes for the corresponding job; Determining, based on the preset workflow template, each group of subtasks, and the number of concurrent nodes of each job, a target working node for each subtask for each job from at least two working nodes of each job; Based on the information of the target working node of each subtask for each job and each group of subtasks, the workflow parameters of each subtask in each group of subtasks are generated, wherein the workflow parameters of each subtask also include: information of the target working node of the corresponding subtask for the at least two jobs.

6. The method according to claim 1, characterized in that At least two execution algorithms for the same job are pre-configured on the worker nodes of each job; Generating workflow parameters for the multiple subtasks according to the preset workflow template and the multiple subtasks includes: Obtaining node algorithm configuration parameters for each of the jobs; Determining a target execution algorithm for each job from at least two execution algorithms for each job according to the node algorithm configuration parameters of each job; The workflow parameters of the multiple subtasks are generated according to the preset workflow template, the multiple subtasks and the target execution algorithm information of each job, wherein the workflow parameters of each subtask also include: the target execution algorithm information of the at least two jobs.

7. The method according to claim 1, characterized in that The using the preset workflow execution engine to execute the multiple subtasks in parallel according to the workflow parameters of the multiple subtasks includes: Using the preset workflow execution engine, calling a preset container platform according to the workflow parameters of each subtask, so that the preset container platform uses the container corresponding to each subtask, calls the execution algorithm of each subtask on the working node of the two jobs, and obtains the execution log of each subtask; After each subtask is completed, the preset workflow execution engine is used to perform persistence processing on the execution log of each subtask.

8. The method according to claim 7, characterized in that The method further comprises: After each subtask is completed, the capacity resources of each subtask are recycled using the preset container platform according to a preset container recycling strategy.

9. The method according to claim 2, characterized in that The method further comprises: Calling the API of the preset workflow execution engine according to the task identifiers of the multiple subtasks to obtain the execution status of the multiple subtasks; Calculating the execution progress of the multiple subtasks and the execution progress of the data closed-loop test task according to the execution status of the multiple subtasks; A graph editing engine is used to display the execution status of the multiple subtasks, the execution progress of the multiple subtasks, and the execution progress of the data closed-loop test task through a visual status display interface.

10. The method according to claim 1, characterized in that If the data closed-loop test task meets the preset test trigger condition, before obtaining the preset workflow template for the data closed-loop test task, the method further includes: Using a graph editing engine to obtain an initial workflow template through the workflow configuration interface; Using the graph editing engine to configure at least two job work node components for the initial workflow template; Using the graph editing engine to configure execution configuration algorithms and execution configuration parameters for the work node components of the at least two jobs respectively; A target workflow template is generated according to the execution configuration algorithm and the execution configuration parameters of the work node components of the at least two jobs.

11. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the data closed-loop workflow processing method as described in any one of claims 1 to 10.

Images