A method and system for single-step debugging in a data processing flow

Abstract

The application provides a method and system for single-step debugging in a data processing flow, and belongs to the field of data processing. The method comprises the following steps: receiving a single-step debugging start instruction, mapping the whole flow to a flow instance by analyzing the content of the data processing flow, and mapping the tasks being executed or having been executed in the flow to task instances; executing the current task instance, updating the task instance state and attribute assignment after the execution is completed; interrupting the flow instance after the execution of the task instance is completed; when the returned task instance information is received, the execution state and input and output data of the task instance are visually displayed according to the task instance state; when the next step running instruction is received, the flow instance identifier is input, the next task to be executed of the flow instance is queried, and the task instance is encapsulated and run, and the above steps are repeated to complete the whole flow debugging. The application can realize flow interruption and recovery, real-time visualization of intermediate data, greatly improve the debugging efficiency, and help to quickly locate errors.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for single-step debugging in a data processing flow. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of data processing, especially in big data analytics, AI model training, and automated ETL (Extract, Transform, Load) processes, complex data processing tasks typically consist of multiple steps, such as data cleaning, transformation, aggregation, and loading. These steps may involve different computational logics, external service calls, or distributed computing frameworks (such as Spark and Flink). However, existing data processing systems have the following limitations in debugging and troubleshooting:

[0004] (1) Lack of single-step execution capability

[0005] Most current data processing systems adopt a batch execution mode. Once the process starts, all task nodes will run automatically in a preset order. If an error occurs in a certain step, developers can usually only rely on log files or post-processing analysis tools to locate the problem. They cannot pause a specific step during runtime to check the intermediate results of data processing step by step.

[0006] (2) Intermediate data is not visible

[0007] In traditional data processing workflows, the input and output data of each task node are typically not persistently stored, or are only recorded in the form of unstructured logs. This makes it difficult for developers to intuitively view the data changes at each step during debugging, resulting in inefficient error localization, especially when complex data transformation logic is involved.

[0008] (3) High debugging cost

[0009] Because developers cannot flexibly control the execution progress of the process, they often need to run the entire process repeatedly and troubleshoot problems step by step using logs or breakpoint debugging tools. This approach is not only time-consuming, but also more difficult to debug in distributed environments (such as Hadoop or Kubernetes clusters). Summary of the Invention

[0010] To overcome the shortcomings of the prior art, this invention provides a method and system for single-step debugging in a data processing workflow. By decomposing the workflow into independently controllable task instances, it enables step-by-step execution, interruption, and resumption. Employing a workflow instantiation architecture, it captures the input and output data of each task in real time and dynamically displays the status and intermediate results through a visual interface. Interactive debugging is supported, allowing users to progressively advance the workflow and analyze data in real time, significantly improving the debugging efficiency of complex workflows.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] The first aspect of this invention provides a method for single-step debugging in a data processing flow;

[0013] A method for single-step debugging in a data processing workflow includes:

[0014] S1 receives the single-step debugging start command and confirms that the current process is in single-step debugging state;

[0015] S2, by parsing the data processing flow content, maps the overall flow to a flow instance, and maps the tasks in execution or already executed in the flow to task instances;

[0016] S3, execute the current task instance, and call the corresponding data processing logic according to the task type and configuration. After execution, update the task instance status and assign attribute values.

[0017] S4: After the current task instance completes execution, interrupt the process instance, stop executing subsequent tasks, and return to the process instance;

[0018] S5, when the returned task instance information is received, the execution status and input / output data of the task instance are visualized on the debug canvas according to the task instance status;

[0019] S6. When the instruction to run the next step is received, the process instance identifier is passed in. The next task to be executed for the process instance is queried and encapsulated as a task instance for execution. Steps S3-S5 are repeated until the entire process debugging is completed.

[0020] As a further technical solution, the system receives a single-step debugging start command and confirms that the current process is in single-step debugging mode, including:

[0021] The normal execution mode and single-step debugging mode are distinguished by passing an identifier during the debugging process.

[0022] As a further technical solution, the task instance includes a task instance identifier, a task instance status, task instance input parameters, task instance output parameters, and a process instance.

[0023] As a further technical solution, the process instance includes a process instance identifier, a task list, and a process instance type. The task list is used to define the execution order of tasks, and the process instance type is used to distinguish whether it is in single-step debugging mode.

[0024] As a further technical solution, the execution of the current task instance, and the invocation of corresponding data processing logic according to the task type and configuration, followed by updating the task instance status and attribute assignments after execution, includes:

[0025] By parsing the process content, the first task to be executed is retrieved from the task list and executed, and then encapsulated as a task instance. At this time, the task instance status is "in execution".

[0026] When running a task instance, different data processing logic is invoked based on different task types and configurations;

[0027] After the task instance is completed, change the task instance status to "execution successful" or "execution failed", and complete the assignment of input and output parameters of the task instance.

[0028] As a further technical solution, upon receiving the returned task instance information, the execution status and input / output data of the task instance are visualized on the debug canvas based on the task instance status, including:

[0029] After the process instance is interrupted and returns, the front end receives the task instance information returned by the back end and displays it on the debug canvas according to the completion status of the task instance. Successful execution is green and failure is red. At the same time, the input and output parameters of this component are displayed to facilitate debugging and analysis by the user.

[0030] A second aspect of the present invention provides a system for single-step debugging in a data processing flow.

[0031] A system for single-step debugging in a data processing workflow, comprising:

[0032] The startup module is configured to receive single-step debugging startup commands and confirm that the current process is in single-step debugging mode.

[0033] The instance mapping module is configured to: map the entire process to a process instance by parsing the data processing flow content, and map the tasks that are being executed or have been executed in the process to task instances;

[0034] The task execution module is configured to: execute the current task instance, call the corresponding data processing logic according to the task type and configuration, and update the task instance status and attribute assignment after execution;

[0035] The interrupt and return module is configured to: interrupt the process instance after the current task instance completes execution, stop the execution of subsequent tasks, and return to the process instance;

[0036] The visualization module is configured to: when receiving returned task instance information, visualize the execution status and input / output data of the task instance on the debug canvas according to the task instance status;

[0037] The next step trigger module is configured to: when it receives the instruction to run the next step, pass in the process instance identifier, query the next task to be executed for that process instance, encapsulate it as a task instance and run it, repeating the process of the task execution module, the interruption and return module and the visualization module until the entire process debugging is completed.

[0038] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a single-step debugging method in a data processing flow as described in the first aspect of the present invention.

[0039] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement steps in a single-step debugging method in a data processing flow as described in the first aspect of the present invention.

[0040] The above one or more technical solutions have the following beneficial effects:

[0041] (1) This invention achieves step-by-step execution, interruption and recovery of the process by breaking down the overall data processing flow into independent task instances. Developers can control the execution rhythm of the process according to their needs, pause and check the execution status at each task node, avoiding the inefficient mode of traditional log analysis and effectively improving debugging efficiency.

[0042] (2) This invention, through a clear mapping and state management between process instances and task instances, allows for interruption of the process at any time during single-step debugging. Analysis is performed based on the execution status and data of the current task instance, and the task can be re-executed through a recovery mechanism if necessary, significantly improving the controllability of debugging complex data processing processes. Furthermore, during the execution of each task instance, intermediate data such as input and output parameters are automatically recorded, and the task status and data details are visually displayed on the front-end canvas. This visualization capability allows developers to monitor the input and output changes of each step in real time, quickly locate abnormal steps and data problems, and reduce the time cost of error troubleshooting.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 This is a flowchart of the method in the first embodiment.

[0046] Figure 2 This is a system structure diagram of the second embodiment. Detailed Implementation

[0047] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0049] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0050] Example 1

[0051] This embodiment discloses a method for single-step debugging in a data processing flow;

[0052] like Figure 1 As shown, a method for single-step debugging in a data processing workflow includes:

[0053] Step S1: Receive the single-step debugging start command and confirm that the current process is in single-step debugging state;

[0054] Step S2: By parsing the data processing flow content, the overall flow is mapped to a flow instance, and the tasks in execution or already executed in the flow are mapped to task instances.

[0055] Step S3: Execute the current task instance and call the corresponding data processing logic according to the task type and configuration. After execution, update the task instance status and assign attribute values.

[0056] Step S4: After the current task instance completes execution, interrupt the process instance, stop executing subsequent tasks, and return to the process instance;

[0057] Step S5: When the returned task instance information is received, the execution status and input / output data of the task instance are visualized on the debug canvas according to the task instance status.

[0058] Step S6: When the instruction to run the next step is received, the process instance identifier is passed in. The next task to be executed for the process instance is queried and encapsulated as a task instance for execution. Steps S3-S5 are repeated until the entire process debugging is completed.

[0059] Specifically, it also includes the following:

[0060] In step S1, to distinguish between normal process startup and single-step debugging startup, a necessary identifier needs to be passed during single-step debugging to indicate that the process is being started in single-step debugging mode. This identifier is the startup mode identifier; for example, 1 represents normal operation, not single-step debugging, and 2 represents single-step debugging. Different processing is performed based on different identifiers.

[0061] In step S2, to facilitate process debugging, the entire process is mapped to a process instance by parsing the data processing process content, and tasks in progress or already executed within the process are mapped to task instances. The main attributes of a task instance include: task instance identifier, task instance status, task instance input parameters, task instance output parameters, and process instance.

[0062] The task instance identifier serves as a unique identifier for each task instance, facilitating logical isolation from other task instances. The task instance status includes the execution status of the task instance, such as successful or failed execution. The task instance input parameters represent the input parameters required to execute this task instance. The task instance output parameters represent the output parameters after the task instance has completed execution.

[0063] A process instance represents the process to which this task instance belongs. The main attributes of a process instance are: process instance identifier, task list, and process instance type. Among them, the process instance identifier is a unique identifier for the process instance, which facilitates logical isolation from other process instances; the task list indicates the execution order of tasks in this process instance; and the process instance type is used to distinguish whether the process instance is in single-step debugging mode.

[0064] In step S3, after single-step debugging starts, the system automatically parses the process content, retrieves the first task to be executed from the task list, and encapsulates it as a task instance. At this time, the task instance status is "Executing". When running the task instance, different data processing logic is invoked according to different task types and configurations. After the task instance is executed, the task instance status is changed to "Execution Successful" or "Execution Failed", and the input parameters, output parameters, and other attributes of the task instance are assigned values.

[0065] In step S4, in single-step debugging mode, a task instance is generated and executed only once at a time. After the current task instance is completed, no subsequent tasks are executed. The process instance is immediately interrupted and returned.

[0066] In step S5, after the process instance is interrupted and returns, the front end receives the task instance information returned by the back end and displays it on the debug canvas according to the completion status of the task instance: green for successful execution and red for failure. Simultaneously, the input and output parameters of this component are also displayed to facilitate debugging and analysis by the user.

[0067] In step S6, when a command to run the next step is received, a process instance identifier is passed in. The next task to be executed for that process instance is queried, encapsulated as a task instance, and run. Steps S3-S5 are repeated until the next task instance is executed and displayed on the debug canvas. This process is repeated until the debugging of the entire process is completed.

[0068] Example 2

[0069] This embodiment discloses a system for single-step debugging in a data processing flow;

[0070] like Figure 2 As shown, a system for single-step debugging in a data processing workflow includes:

[0071] The startup module is configured to receive single-step debugging startup commands and confirm that the current process is in single-step debugging mode.

[0072] The instance mapping module is configured to: map the entire process to a process instance by parsing the data processing flow content, and map the tasks that are being executed or have been executed in the process to task instances;

[0073] The task execution module is configured to: execute the current task instance, call the corresponding data processing logic according to the task type and configuration, and update the task instance status and attribute assignment after execution;

[0074] The interrupt and return module is configured to: interrupt the process instance after the current task instance completes execution, stop the execution of subsequent tasks, and return to the process instance;

[0075] The visualization module is configured to: when receiving returned task instance information, visualize the execution status and input / output data of the task instance on the debug canvas according to the task instance status;

[0076] The next step trigger module is configured to: when it receives the instruction to run the next step, pass in the process instance identifier, query the next task to be executed for that process instance, encapsulate it as a task instance and run it, repeating the process of the task execution module, the interruption and return module and the visualization module until the entire process debugging is completed.

[0077] Specifically, in the instance mapping module, the task instance includes a task instance identifier, task instance status, task instance input parameters, task instance output parameters, and process instance. The process instance includes a process instance identifier, a task list, and a process instance type. The task list is used to define the execution order of tasks, and the process instance type is used to distinguish whether it is in single-step debugging mode.

[0078] Example 3

[0079] The purpose of this embodiment is to provide a computer-readable storage medium.

[0080] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a step-by-step debugging method in a data processing flow as described in Embodiment 1.

[0081] Example 4

[0082] The purpose of this embodiment is to provide an electronic device.

[0083] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a single-step debugging method in a data processing flow as described in Embodiment 1.

[0084] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0085] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0086] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for single-step debugging in a data processing workflow, characterized in that, include: S1 receives the single-step debugging start command and confirms that the current process is in single-step debugging state; S2, by parsing the data processing flow content, the overall flow is mapped to a flow instance, and the tasks in execution or already executed in the flow are mapped to task instances; the task instance includes a task instance identifier, task instance status, task instance input parameters, task instance output parameters, and flow instance; the flow instance includes a flow instance identifier, a task list, and a flow instance type, the task list is used to define the execution order of the tasks, and the flow instance type is used to distinguish whether it is in single-step debugging state. S3, execute the current task instance, and call the corresponding data processing logic according to the task type and configuration. After execution, update the task instance status and assign attribute values. S4: After the current task instance completes execution, interrupt the process instance, stop executing subsequent tasks, and return to the process instance; S5, when receiving the returned task instance information, visualizes the execution status and input / output data of the task instance on the debug canvas according to the task instance status. Specifically, after the process instance is interrupted and returns, the front end receives the task instance information returned by the back end and displays it on the debug canvas according to the completion status of the task instance. Successful execution is green and failed execution is red. At the same time, the input and output parameters of this component are displayed to facilitate users' debugging and analysis. S6. When the instruction to run the next step is received, the process instance identifier is passed in. The next task to be executed for the process instance is queried and encapsulated as a task instance for execution. Steps S3-S5 are repeated until the entire process debugging is completed.

2. The method for single-step debugging in a data processing flow as described in claim 1, characterized in that, Receive the single-step debugging start command and confirm that the current process is in single-step debugging mode, including: The normal execution mode and single-step debugging mode are distinguished by passing an identifier during the debugging process.

3. The method for single-step debugging in a data processing flow as described in claim 1, characterized in that, The process of executing the current task instance, invoking the corresponding data processing logic according to the task type and configuration, and updating the task instance status and attribute values ​​after execution includes: By parsing the process content, the first task to be executed is retrieved from the task list and executed, and then encapsulated as a task instance. At this time, the task instance status is "executing". When running a task instance, different data processing logic is invoked based on different task types and configurations; After the task instance is completed, change the task instance status to "execution successful" or "execution failed", and complete the assignment of input and output parameters of the task instance.

4. A system for single-step debugging in a data processing workflow, characterized in that, include: The startup module is configured to receive single-step debugging startup commands and confirm that the current process is in single-step debugging mode. The instance mapping module is configured to: map the entire process to a process instance by parsing the data processing flow content, and map the tasks in execution or already executed in the process to task instances; the task instance includes a task instance identifier, task instance status, task instance input parameters, task instance output parameters, and process instance; the process instance includes a process instance identifier, a task list, and a process instance type, the task list is used to define the execution order of tasks, and the process instance type is used to distinguish whether it is in single-step debugging state; The task execution module is configured to: execute the current task instance, call the corresponding data processing logic according to the task type and configuration, and update the task instance status and attribute assignment after execution; The interrupt and return module is configured to: interrupt the process instance after the current task instance has completed execution, stop the execution of subsequent tasks, and return to the process instance; The visualization module is configured to: when receiving returned task instance information, visually display the execution status and input / output data of the task instance on the debug canvas according to the task instance status; specifically, after the process instance is interrupted and returns, the front end receives the task instance information returned by the back end and displays it on the debug canvas according to the completion status of the task instance, with green for successful execution and red for failure; at the same time, the input and output parameters of this component are also displayed to facilitate debugging and analysis by the user; The next step trigger module is configured to: when it receives the instruction to run the next step, pass in the process instance identifier, query the next task to be executed for that process instance, encapsulate it as a task instance and run it, repeating the process of the task execution module, the interruption and return module and the visualization module until the entire process debugging is completed.

5. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the single-step debugging method in a data processing flow as described in any one of claims 1-3.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the single-step debugging method in a data processing flow as described in any one of claims 1-3.

Images