Data development debugging method and device, equipment, storage medium and program product

Abstract

The invention provides a data development debugging method and device, equipment, a storage medium and a program product. Relates to the technical field of data development. The method comprises the following steps: in response to a received debugging request initiated by a user through a data development platform, allocating computing resources based on the debugging request and starting an interactive debugging environment instance; the debugging request comprises a Python operator code to be debugged; executing the Python operator code to be debugged in the interactive debugging environment instance, and displaying a code execution result and a variable value; and backfilling the debugged Python operator code in the interactive debugging environment instance into a static code editor of the data development platform. According to the method provided by the invention, the code execution state can be checked in real time, the code writing and debugging process is deeply integrated, environment switching and manual operation are avoided, and human errors are reduced; through a reasonable resource management and scheduling mechanism, it is ensured that computing resources can be fully utilized in the starting and running process of the interactive debugging environment instance, and resource waste is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data development, and particularly relates to a debugging method and device for data development, equipment, a storage medium and a program product. BACKGROUND

[0002] In the field of enterprise data development, the integration and processing requirements of multi-source heterogeneous data are increasingly complex, and developers need to deal with data cleaning, conversion and analysis tasks of various sources such as relational databases, NoSQL and real-time streaming data. As a mainstream data processing language, the development and debugging efficiency of Python operators directly affects the project progress and data quality.

[0003] In the current data development platform, the code editing and debugging links are generally in a fragmented state. Developers need to write Python operator logic in a static code editor first, and after completion, they need to manually package the code and submit it to an independent test environment through a platform deployment tool. After the test environment starts a complete task instance, the code can run, and during this period, the execution process cannot be intervened in real time. Developers can only rely on log files generated after the task is completed, or insert a large number of print statements in the code to print variable values, and locate problems by searching logs line by line. If the code needs to be modified, it needs to be returned to the static editor for adjustment and then redeployed.

[0004] Such a split-state data development environment requires frequent switching between environments and relies on post-log analysis, which increases operational complexity and human error, and has low debugging efficiency. The rough allocation of resources leads to idle and wasted resources. SUMMARY

[0005] The debugging method, device, equipment, storage medium and program product for data development provided by the embodiments of the present application are used to effectively integrate data development code editing and debugging, and improve the debugging efficiency.

[0006] In a first aspect, the embodiments of the present application provide a debugging method for data development, which comprises:

[0007] In response to receiving a debugging request initiated by a user through a data development platform, a computing resource is allocated based on the debugging request and an interactive debugging environment instance is started; the debugging request includes Python operator code to be debugged;

[0008] The Python operator code to be debugged is executed in the interactive debugging environment instance, and the code execution result and variable value are displayed;

[0009] The Python operator code after debugging in the interactive debugging environment instance is backfilled into the static code editor of the data development platform.

[0010] In a possible implementation, the allocating computing resources and starting an interactive debugging environment instance based on the debugging request comprises:

[0011] obtaining a current cluster resource state;

[0012] determining a target computing resource group according to the type of the Python operator code to be debugged and the current cluster resource state;

[0013] creating a corresponding resource object according to the target computing resource group by using a preset container orchestration tool, and creating and starting an interactive debugging environment instance in the target computing resource group.

[0014] In a possible implementation, the Python operator code to be debugged is distributed, and the executing the Python operator code to be debugged in the interactive debugging environment instance and displaying code execution results and variable values comprises:

[0015] obtaining an execution path and data flow information of the Python operator code to be debugged by using a preset distributed tracking framework;

[0016] displaying the execution path and data flow information in the interactive debugging environment instance.

[0017] In a possible implementation, after the backfilling the Python operator code debugged in the interactive debugging environment instance into the static code editor of the data development platform, the method further comprises:

[0018] releasing corresponding computing resources according to a preset recycling rule;

[0019] the preset recycling rule comprises:

[0020] releasing computing resources corresponding to the interactive debugging environment instance in response to a running time of the interactive debugging environment instance reaching a preset running time length;

[0021] releasing computing resources corresponding to the interactive debugging environment instance in response to a resource occupancy rate of the interactive debugging environment instance being less than a preset occupancy rate threshold.

[0022] In a possible implementation, after the backfilling the Python operator code debugged in the interactive debugging environment instance into the static code editor of the data development platform, the method further comprises:

[0023] obtaining environment configuration information of the interactive debugging environment instance and the data development platform;

[0024] generating an environment difference report based on the environment configuration information;

[0025] In response to the environment difference report, synchronize the missing configuration item in the environment of the data development platform.

[0026] In a possible implementation, the method further includes:

[0027] If it is determined that the memory usage rate continuously exceeds the preset usage rate threshold for a first preset time duration during code execution, apply for temporary expansion of memory resources to the target computing resource group;

[0028] If it is determined that the code execution is in an idle state for a second preset time duration, release a preset proportion of idle CPU resources.

[0029] In a possible implementation, before the computing resource is allocated and the interactive debugging environment instance is started based on the debugging request, the method further includes:

[0030] Synchronize the Python operator code to be debugged to the interactive debugging environment instance based on a preset synchronization mechanism.

[0031] In a second aspect, an embodiment of the present application provides a debugging device for data development, including:

[0032] The allocation and starting module is configured to, in response to receiving a debugging request initiated by a user through a data development platform, allocate a computing resource and start an interactive debugging environment instance based on the debugging request; the debugging request includes Python operator code to be debugged.

[0033] The execution and display module is configured to execute the Python operator code to be debugged in the interactive debugging environment instance, and display code execution results and variable values.

[0034] The backfilling module is configured to backfill the Python operator code after debugging in the interactive debugging environment instance to a static code editor of the data development platform.

[0035] In a third aspect, an embodiment of the present application provides a debugging device for data development, including a memory and a processor.

[0036] The memory stores computer execution instructions.

[0037] The processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementations of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0040] The debugging method, device, equipment, storage medium and program product for data development provided by the embodiments of the present application can be used in a data development platform. A user submits a debugging request, and the request contains Python operator code to be debugged. After receiving the request, computing resources are allocated based on requirements, and an interactive debugging environment instance is started. Not only is the waste of resources or excessive occupation avoided, but also the debugging function is deeply integrated with the data development platform. There is no need to switch between multiple tools and environments, the development process is simplified, and the development efficiency is improved. The code is executed in the interactive debugging environment instance, and the execution result is displayed. The user can understand the code running situation at the first time, and quickly locate the problem. The variable value is displayed to observe the change of the variable in the code execution process in real time, so that the code logic can be more accurately analyzed, and the potential problem source can be found. The interactive debugging environment allows the user to perform dynamic operations in the code execution process, such as setting breakpoints and single-step execution. The debugging process can be more flexibly controlled, the problem can be gradually checked, and the efficiency and accuracy of debugging are greatly improved. The Python operator code after debugging is backfilled into the static code editor of the data development platform, and the consistency of the code in different environments is ensured. The code backfilled into the static code editor can be conveniently managed and reviewed. The modification history of the code is managed through the version control system, and the quality and maintainability of the code are ensured. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0042] Figure 1 An application scenario diagram of a debugging method for data development provided by the present application;

[0043] Figure 2 A flowchart of a debugging method for data development provided by an embodiment of the present application;

[0044] Figure 3 A flowchart of a debugging method for data development provided by another embodiment of the present application;

[0045] Figure 4 A structural schematic diagram of a debugging device developed by data provided for an embodiment of the present application is shown in the figure.

[0046] Figure 5 A structural schematic diagram of a debugging device developed by data provided for an embodiment of the present application is shown in the figure.

[0047] The specific embodiments of the present application have been shown in the above figures, and will be described in more detail hereinafter. These figures and the textual description are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0048] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context of use indicates otherwise. The following exemplary embodiments described in the following examples are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0049] In the technical solution of the present application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data, etc. comply with relevant laws and regulations and do not violate public order and good customs.

[0050] It should be noted that in the embodiments of the present application, some existing industry solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0051] In order to clearly understand the technical solution of the present application, the prior art solution will be described in detail first.

[0052] In the field of enterprise data development, as the business develops, the integration and processing of multi-source heterogeneous data are increasingly complex. Developers often face data cleaning, conversion, and analysis from various data sources such as relational databases, NoSQL databases, and real-time streaming data. Python has become the mainstream language for data processing due to its powerful features and rich libraries. The development and debugging efficiency of Python operators directly affects the project progress speed and data quality. Most current data development platforms have a clear separation in code editing and debugging. Developers usually write the logic code of Python operators in a static code editor first. After writing, they manually package the code and submit it to the independent test environment using the platform's deployment tools. Only after the test environment starts a complete task instance can the code start running. During code execution, developers cannot intervene in real time and can only passively wait for the task to end. After the task ends, developers can only rely on generated log files to troubleshoot problems or insert a large number of print statements in the code to print variable values, then locate the problem by viewing the log line by line. If the code needs to be modified, developers need to go back to the static editor to make adjustments, and then repackage and deploy. This split data development environment requires developers to frequently switch between different environments, relies too much on post-log analysis, not only increases the complexity of operations, but also easily causes human errors, resulting in very low debugging efficiency. Moreover, there is a lack of fine management of resource allocation, often resulting in unreasonable resource allocation and waste of resources.

[0053] Therefore, in the face of technical problems in the prior art, to eliminate the fragmentation of the development process and reduce the cost of environment switching, a debugging request response mechanism is designed. After receiving a debugging request from the data development platform, computing resources are allocated and a debugging environment instance is started, realizing seamless connection between development and debugging environments, and avoiding manual transfer of code between multiple tools. To solve the problem of waste caused by extensive resource allocation, computing resources are allocated dynamically based on debugging requests, only necessary resources are occupied during debugging, and invalid consumption is reduced. To solve the problem of low debugging efficiency and reliance on post-log analysis, the Python operator code to be debugged is executed in real time in the interactive debugging environment instance and the execution results and variable values are dynamically displayed, avoiding the lag of traditional log troubleshooting, accurately locating the problem node, and improving the debugging efficiency. To ensure that the debugging results can directly affect the development process, a code backfill strategy is adopted to automatically synchronize the code after debugging in the interactive environment instance to the static editor of the data development platform, avoiding version inconsistency problems.

[0054] Figure 1 The application scenario diagram of the data development debugging method provided by the present application is as follows, Figure 1As shown in the diagram, the scenario corresponding to the data development debugging method provided in this application includes: a terminal device 101 and a server 102. The data development debugging device can be integrated into the server 102, which is the core device for deploying the data development platform.

[0055] Specifically, users can initiate a debugging request by dragging the Python operator code to be debugged into the debugging input box and clicking the debug button in the visual interface of terminal device 101. After receiving the debugging request initiated by the user through the data development platform, server 102 allocates computing resources and starts an interactive debugging environment instance based on the debugging request. Server 102 executes the Python operator code to be debugged in the interactive debugging environment instance. At this time, the service status of the interactive debugging environment instance, code execution results, and variable values ​​will be displayed in the visual interface of terminal device 101. If debugging of the Python operator code is required, users can directly modify it on the interface displayed on terminal device 101. Server 102 populates the debugged Python operator code in the interactive debugging environment instance back into the static code editor of the data development platform for users to perform subsequent operations.

[0056] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0057] Figure 2 A flowchart illustrating a debugging method for data development provided in an embodiment of this application is shown below. Figure 2 As shown, the execution entity in this embodiment is a data development debugging device. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc, or through a physical device integrating or installing the relevant computer program, such as a chip or data development debugging equipment. The data development debugging equipment can be a server, server cluster, etc. The data development debugging method provided in this embodiment includes the following steps:

[0058] S201. In response to receiving a debugging request initiated by a user through the data development platform, allocate computing resources based on the debugging request and start an interactive debugging environment instance; the debugging request includes the Python operator code to be debugged.

[0059] Among them, a debug request refers to an instruction triggered by the user in the data development platform to start the debugging process. It can include the Python operator code to be debugged and debugging parameters, such as the running environment and resource requirements.

[0060] Interactive debugging environment instances refer to temporary debugging environments created using tools such as Jupyter Notebook, which support running code and viewing results in real time.

[0061] Specifically, after writing Python operator code in the static code editor of the data development platform, the user clicks the "Start Debugging" button. The platform automatically encapsulates the debugging request, including the operator code and user-configured debugging parameters. Then, the data development platform sends the debugging request to the data development debugging device through a front-end interface, such as HTTP / HTTPS. The request can carry unique identifiers, such as task ID, operator code text, and resource requirement parameters.

[0062] Furthermore, after receiving a debugging request, the data development debugging device queries the current cluster resource status through a cluster monitoring interface, such as the Kubernetes API, including the CPU, memory, GPU utilization, and distribution of idle resource pools for each node. If the operator code in the debugging request involves machine learning or other GPU-intensive scenarios, nodes with sufficient GPU resources are prioritized; if the operator is distributed processing logic, a resource group with a sufficient number of nodes is selected based on the parallelism parameters defined in the code; finally, the target resource group is determined.

[0063] Furthermore, the data development debugging apparatus uses container orchestration tools, such as Kubernetes, to create container instances (Pods) in the target resource group, and to create resources such as Deployments, Services, and Ingresses. It automatically configures the container's network parameters, such as internal IP and port, and generates externally accessible URLs through the Ingress controller.

[0064] Understandably, the debugging tool for data development will automatically write the Python operator code to be debugged in the debugging request into the initial cell of the Jupyter Notebook. Users can directly see the code to be debugged after opening the URL, without having to upload it manually.

[0065] S202. Execute the Python operator code to be debugged in the interactive debugging environment instance, and display the code execution results and variable values.

[0066] Specifically, after the interactive environment instance starts, it loads the Python operator code from the debug request into the code cell and initializes the runtime environment, such as importing libraries that the code depends on and connecting to the data source required for debugging. Users can execute the operator logic line by line or block by block by clicking the run cell button.

[0067] Optionally, users can set breakpoints at key locations in the code, such as loops and conditional statements, and the code will automatically pause when the breakpoint is reached. It also supports operations such as stepping over (next) and stepping into (step).

[0068] Optionally, for print statements, the text result can be displayed immediately below the code cell after execution; for data structures, they can be automatically converted into tables or charts through a visualization component, supporting paginated viewing and data filtering.

[0069] Optionally, if an exception occurs during execution, such as a syntax error or data type mismatch, the error location will be displayed and detailed error information will be output to help locate the problem.

[0070] Optionally, all variables in the current scope, such as df and count, can be listed directly on the interface, along with their types, such as DataFrame and int. Users can also click on a variable name to view details, such as expanding the key-value pairs of a dictionary, viewing the complete elements of a list, or actively searching for the variable in a code cell by entering its name.

[0071] S203. Backfill the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform.

[0072] Among them, the debugged Python operator code refers to the Python data processing code that has been modified and verified by the user in an interactive debugging environment.

[0073] Among them, the static code editor refers to the basic tool used for writing code in the data development platform, which supports functions such as code editing and syntax highlighting.

[0074] Specifically, the data development debugging device monitors the code modification status within the interactive debugging environment instance, recording user modifications to the operator code in real time, such as adding, deleting, and replacing lines of code, and generating a change log containing code snippets before and after modification, along with modification timestamps. After the user debugs the Python operator code, the data development debugging device encapsulates the modified code in a format agreed upon with the data development platform, such as JSON, and transmits it to the platform's backend service via an encrypted interface, such as HTTPS. The platform's backend then pushes the debugged code to the static code editor, overwriting the original code to be debugged in the editor.

[0075] Optionally, a message indicating that the code has been backfilled from the debugging environment can be displayed at the top of the static code editor, along with a button to view the change history. Users can click to view the specific modifications made during this debugging session, such as comparing the differences between the code before and after debugging.

[0076] Optionally, before backfilling, by comparing the differences between the code in the debugging environment and the original code in the static editor, if the original code was modified by other users during debugging, a conflict warning is triggered to avoid overwriting other people's modifications.

[0077] The data development debugging method provided in this application embodiment allows users to submit debugging requests within a data development platform. These requests include Python operator code to be debugged. Upon receiving the request, the platform allocates computing resources based on demand and launches an interactive debugging environment instance. This not only avoids idle or excessive resource usage but also deeply integrates debugging functionality with the data development platform, eliminating the need to switch between multiple tools and environments, simplifying the development process and improving efficiency. Executing code and displaying execution results in the interactive debugging environment instance allows users to understand the code's running status immediately and quickly locate problems. Displaying variable values ​​allows for real-time observation of changes in variables during code execution, enabling more accurate analysis of code logic and identification of potential root causes. The interactive debugging environment allows users to perform dynamic operations during code execution, such as setting breakpoints and single-stepping. This provides more flexible control over the debugging process, allowing for step-by-step troubleshooting and significantly improving debugging efficiency and accuracy. The debugged Python operator code is then backfilled into the static code editor of the data development platform, ensuring code consistency across different environments. The code backfilled into the static code editor facilitates version management and code review. By using a version control system to track and manage the code's modification history, we can ensure the quality and maintainability of the code.

[0078] As an optional implementation, based on the above embodiments, allocating computing resources and launching an interactive debugging environment instance based on a debugging request includes:

[0079] Get the current cluster resource status;

[0080] The target computing resource group is determined based on the type of Python operator code to be debugged and the current cluster resource status;

[0081] The pre-defined container orchestration tool creates corresponding resource objects based on the target compute resource group, and creates and starts an interactive debugging environment instance in the target compute resource group.

[0082] Among them, cluster resource status refers to the usage and distribution of hardware resources of each node in the computing cluster where the data development platform is located, such as a Kubernetes cluster.

[0083] Among them, the Python operator code type to be debugged refers to the category divided according to the code function and running requirements. For example: lightweight, which only involves simple data cleaning and has low resource requirements; computationally intensive, which contains complex mathematical operations and requires GPU support; and distributed, which requires multi-node collaborative processing and depends on the number of cluster nodes.

[0084] The target computing resource group refers to a set of computing nodes selected from the cluster that meet the debugging requirements. It is the specific location for allocating resources and deploying the debugging environment.

[0085] Among them, pre-configured container orchestration tools refer to pre-configured tools for managing the lifecycle of containers, such as Kubernetes, which can automatically create, schedule, and destroy container instances.

[0086] Resource objects refer to entities defined in container orchestration tools that describe resource allocation and service configuration, such as Deployment, Service, and Ingress in Kubernetes.

[0087] Specifically, resource data for each node, such as hardware resources, node status, and resource group attributes, is collected in real time through cluster monitoring interfaces, such as Kubernetes' Metrics API. The collected, scattered data is aggregated into a structured resource status table and displayed categorized by resource group. Based on the type of Python operator code to be debugged, the group with the highest overall resource sufficiency is selected from the matching resource groups as the target computing resource group.

[0088] For example, for lightweight code, prioritize general resource groups with sufficient idle CPU or memory and low load, such as test resource groups; for computationally intensive code, select resource groups that include available GPUs, such as GPU-dedicated groups, and ensure that the memory capacity meets the model's running requirements; for distributed code, select resource groups with a number of nodes greater than or equal to the code parallelism parameter and a network latency between nodes less than 10ms, such as distributed computing groups.

[0089] Furthermore, pre-configured container orchestration tools are used to create resource objects. Taking Kubernetes as an example, Deployment pre-installs a customized image containing Jupyter Notebook, Python, and data processing libraries; CPU, memory, and GPU request limits and caps are set according to code type. Service assigns fixed internal access addresses to container instances, enabling the cluster's scheduling system and log service to stably access the debugging environment. Ingress configures external access rules, forwarding user data development platform requests to Service via domain names, ultimately mapping to the container's Jupyter port. Based on the Deployment configuration, the pre-configured container orchestration tool creates and starts interactive debugging environment instances on the nodes of the target compute resource group, automatically pulling images and initializing the environment.

[0090] The data development debugging method provided in this application obtains the current cluster resource status, enabling a clear understanding of the usage and remaining amount of various resources in the cluster. Based on the type of Python operator code to be debugged and the current cluster resource status, the target computing resource group is determined, allowing for precise allocation of resources to the debugging task. This avoids idle waste caused by excessive resource allocation and prevents debugging tasks from failing to execute smoothly or experiencing performance degradation due to insufficient resource allocation. A preset container orchestration tool is used to create corresponding resource objects based on the target computing resource group, and an interactive debugging environment instance is created and started within the target computing resource group. This enables rapid creation of resource objects and environment startup, significantly shortening the preparation time of the debugging environment, reducing waiting time, and improving debugging efficiency.

[0091] As an optional implementation, based on the above embodiments, the Python operator code to be debugged is distributed. The Python operator code to be debugged is executed in an interactive debugging environment instance, and the code execution results and variable values ​​are displayed, including:

[0092] A pre-defined distributed tracing framework is used to obtain the execution path and data flow information of the Python operator code to be debugged;

[0093] The execution path and data flow information are displayed in the interactive debugging environment instance.

[0094] Distributed Python operator code refers to Python data processing code that needs to run collaboratively on multiple nodes.

[0095] Among them, the pre-defined distributed tracing framework refers to a tool that is pre-integrated into the debugging environment to trace the execution process of distributed code on multiple nodes, and record the call relationship, data transmission path and execution status of each node.

[0096] The execution path refers to the flow of distributed code across multiple nodes, including the task allocation order, function call relationships, and execution status of each node.

[0097] Among them, data flow information refers to the details of data transmission between nodes in distributed computing, including the source node, target node, transmission time, data size, and key field values ​​of data shards.

[0098] Specifically, when the interactive debugging environment instance starts, it automatically loads a pre-defined distributed tracing framework, such as a tracing component based on the OpenTelemetry protocol, and injects tracing probes into the distributed Python operator code to be debugged. The probes automatically generate tracing identifiers and span identifiers at key code nodes, such as task submission, node communication, and data sharding, to associate the execution logs of different nodes.

[0099] Furthermore, when the code runs on multiple nodes of a distributed cluster, the tracing framework synchronizes tracing identifiers through inter-node communication protocols, such as RPC, and records the start time, task type, and dependencies of each node, such as node B's task depending on the output of node A. The tracing framework intercepts data transmission operations between nodes, recording the source node IP, target node IP, transmission start / end time, data volume, and core metadata.

[0100] Furthermore, the interactive environment instance converts the traced execution path into a node relationship graph, using circular nodes to represent physical nodes in the cluster, arrows to indicate the direction of task invocation, and annotations next to the arrows indicating the invocation time and execution status. Data flow information between nodes is displayed in the form of a data flow graph.

[0101] Optionally, if a node fails to execute or a data transmission times out, the corresponding node / transmission line will be automatically highlighted and the reason for the error will be displayed. At the same time, the last variable value and stack information of the node will be displayed to help quickly troubleshoot cross-node problems in distributed scenarios.

[0102] The data development debugging method provided in this application uses a pre-defined distributed tracing framework to obtain execution path and data flow information. This allows developers to clearly see the execution order and call relationships of the code on each node, quickly locating the key aspects of code execution. This avoids blindly troubleshooting across multiple nodes, significantly shortening debugging time and improving debugging efficiency. In a distributed system, a problem may involve the collaborative work of multiple nodes. When errors or exceptions occur during code execution, the execution path information obtained through the distributed tracing framework can quickly identify the specific node where the problem occurred. This avoids a comprehensive investigation of the entire distributed system, improving the accuracy and efficiency of problem localization.

[0103] As an optional implementation, based on the above embodiments, after backfilling the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform, the method further includes:

[0104] Release the corresponding computing resources according to the preset recycling rules;

[0105] The preset recycling rules include:

[0106] In response to the interactive debugging environment instance reaching the preset runtime, the computing resources corresponding to the interactive debugging environment instance are released.

[0107] In response to the resource utilization rate of the interactive debugging environment instance being less than the preset utilization rate threshold, the computing resources corresponding to the interactive debugging environment instance are released.

[0108] Among them, the preset recycling rules refer to the set of conditions pre-configured by the system to determine when to release computing resources in the debugging environment.

[0109] The runtime of the interactive debugging environment instance refers to the duration from the start of the debugging environment instance to the present.

[0110] The preset runtime refers to the maximum allowed running time threshold of the pre-set debugging environment, such as 1 hour.

[0111] Resource utilization rate refers to the ratio of resources currently used by the interactive debugging environment instance to the allocated resources.

[0112] Among them, the preset utilization threshold refers to the pre-set criteria for judging low resource load, such as CPU utilization < 10% and memory utilization < 15%.

[0113] Specifically, once the debugged Python operator code is successfully populated back into the static code editor of the data development platform, a resource reclamation monitoring process is initiated to track the running status of the interactive debugging environment instance in real time. The runtime and resource utilization of the debugging environment instance are periodically collected using cluster monitoring tools.

[0114] For example, computing resources can be released according to preset reclamation rules. The current runtime is compared with the preset runtime, and resource reclamation is triggered when the runtime is greater than or equal to the preset value. Alternatively, if the CPU utilization rate of the debugging environment is detected to be less than a preset threshold for a continuous period of time, it is determined that the resources are idle, and resource reclamation is triggered.

[0115] Optionally, if the runtime environment reaches the preset duration, a notification can be pushed to the user's data development platform before releasing resources. For example, the notification might state that the debugging environment will be released in 30 seconds, and users should click to extend the duration if they wish to continue using it. If the user does not take any action, a termination command will be sent to the target instance via the container orchestration tool after 30 seconds. After the instance terminates, the corresponding container resources are automatically deleted, and the occupied CPU, memory, and other computing resources are released to the cluster resource pool.

[0116] The data development debugging method provided in this application allows for the release of corresponding computing resources when the interactive debugging environment instance reaches a preset runtime. This indicates that the debugging task may have been completed or has not made progress for a long time. This prevents these resources from being continuously occupied and idle, allowing them to flow back to the resource pool for use by other tasks that require them. If the resource utilization rate of the interactive debugging environment instance is less than a preset utilization rate threshold, it indicates that the current instance has a low demand for resources. After releasing these resources, they can be reallocated to tasks requiring higher configurations based on new task needs, achieving dynamic optimization of resource allocation and improving overall resource utilization efficiency.

[0117] As an optional implementation, based on the above embodiments, after backfilling the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform, the method further includes:

[0118] Obtain the environment configuration information of the interactive debugging environment instance and the data development platform;

[0119] Generate an environmental difference report based on environmental configuration information;

[0120] Based on the environment difference report, synchronize missing configuration items in the data development platform environment.

[0121] Among them, the environment configuration information refers to the set of runtime environment parameters for the interactive debugging environment instance and the data development platform, including but not limited to Python version, dependency library version, and system environment variables such as data storage path and authentication key.

[0122] The environment difference report is a document generated by comparing the configuration information of the debugging environment and the development platform. It is used to record the inconsistencies in the configuration items between the two and to mark the types of differences.

[0123] The missing configuration items refer to key parameters that exist in the interactive debugging environment instance but are not configured in the data development platform, such as the debugging environment depending on PySpark 3.4.0 while the development platform only has 3.2.0 installed.

[0124] Specifically, after code refilling, a scanning tool scans the environment configuration information of the interactive debugging environment instance, such as dependency library information, environment variables, and framework configuration parameters. The environment configuration information of the data development platform, such as the project dependency list, global environment variables, and deployment configuration, is extracted through the configuration management interface.

[0125] Furthermore, the environment configuration information is compared item by item. For example, if a library exists in the debugging environment but not in the development platform, it is marked as a missing debugging dependency on the development platform; if the versions are inconsistent, they are marked as incompatible. The differences are compiled into a report, which may include difference ID, category, debugging environment configuration, development platform configuration, etc.

[0126] Furthermore, synchronization operations are performed based on the environmental difference report, such as automatically installing or upgrading dependency libraries through the development platform's package management tools.

[0127] Optionally, after synchronization is complete, the development platform configuration is re-collected and compared with the debugging environment to confirm that the differences have been corrected; the synchronization log is displayed on the development platform interface, and if synchronization fails, such as due to insufficient permissions, the user is prompted with a highlight and provided with manual operation guidance.

[0128] The data development debugging method provided in this application embodiment addresses the possibility that the interactive debugging environment and the data development platform may have different software versions, dependency library configurations, etc. By acquiring the environment configuration information of both and generating a difference report, the differences can be accurately located. By synchronizing missing or inconsistent configuration items, it can be ensured that the environment on which the code depends is consistent with the debugging environment when the code runs on the data development platform, thereby avoiding runtime errors caused by environment incompatibility.

[0129] As an optional implementation, based on the above embodiments, it further includes:

[0130] If it is determined during code execution that the memory usage rate exceeds the preset usage rate threshold for a first preset duration, then a temporary expansion of memory resources will be requested from the target computing resource group.

[0131] If the duration of idle code execution reaches a second preset duration, then a preset proportion of idle CPU resources will be released.

[0132] Memory utilization rate refers to the ratio of the amount of memory currently used by the interactive debugging environment instance to the total amount of memory allocated.

[0133] The first preset duration refers to a pre-set time threshold used to determine whether memory usage continues to exceed the limit, such as 5 seconds.

[0134] The preset utilization threshold refers to the critical value that triggers memory expansion, such as 80%. If this value is exceeded, it is determined that memory resources are insufficient.

[0135] The idle state refers to a state in which no effective operation is performed during code execution.

[0136] The second preset duration refers to a pre-set time threshold for determining whether a period of long-term idle time has occurred, such as 2 minutes.

[0137] The preset percentage of idle CPU resources refers to the pre-set percentage of CPU resources that can be released when idle, such as 50%.

[0138] For example, the cluster monitoring tool collects memory usage, CPU usage, and operation logs of the interactive debugging environment instance. If memory usage is greater than or equal to a preset threshold for a first preset duration, it is marked as memory strained. Based on the current memory usage and code type, a temporary capacity expansion is determined; a memory expansion request is sent to the target compute resource group via the container orchestration tool API. The container orchestration tool dynamically adjusts the memory limit of the debugging environment instance without restarting the instance. If CPU usage is less than or equal to 5% for a second preset duration, and there are no user interactions, it is marked as idle. CPU resources are released according to a preset ratio.

[0139] Optionally, when a user restarts the debugging environment, if a sudden increase in CPU usage is detected, it is determined to be a return to an active state; and the system automatically requests the target computing resource group to restore the original CPU allocation.

[0140] The data development debugging method provided in this application allows for dynamic changes in memory and CPU resource requirements among different tasks during code execution. When a first preset duration exceeds a preset utilization threshold, temporary memory expansion is requested to promptly provide the necessary memory for the task, ensuring smooth execution and preventing task blocking or failure due to insufficient memory. Similarly, when code execution is idle for a second preset duration, a preset proportion of idle CPU resources is released, freeing these resources for other tasks, achieving precise resource matching and efficient utilization.

[0141] As an optional implementation, based on the above embodiments, before allocating computing resources based on the debugging request and starting the interactive debugging environment instance, the following further steps are included:

[0142] Based on the preset synchronization mechanism, the Python operator code to be debugged is synchronized to the interactive debugging environment instance.

[0143] Among them, the preset synchronization mechanism refers to the standardized process that the system pre-configures for transferring the code to be debugged from the data development platform to the interactive debugging environment instance, such as Git version control or rsync file synchronization protocol.

[0144] Optionally, before synchronization, the integrity of the code to be debugged can be checked, such as whether there are any unsaved modifications or syntax errors. If there are unsaved contents, the user will be automatically prompted to save the code before starting debugging; if there are syntax errors, the location of the error will be displayed, and synchronization will be performed after the user corrects it.

[0145] Specifically, the system automatically selects a preset synchronization mechanism based on the code size and scenario. For example, small-scale code uses HTTP / HTTPS protocol to directly transmit code text, while large-scale or multi-file code uses Git version control protocol for synchronization.

[0146] Optionally, if a code file with the same name already exists in the debugging environment instance, the system will process it according to preset rules. If the local file has not been modified, it will be overwritten directly; if the local file has been modified and not backed up to the platform, the local file will be automatically renamed and backed up, and then the latest code on the platform will be synchronized.

[0147] The data development debugging method provided in this application addresses the issue that in projects with multi-person collaborative development or frequent code updates, different developers may modify the code at different times, easily leading to code version inconsistencies. By using a preset synchronization mechanism to synchronize the code to be debugged to an interactive debugging environment instance, it ensures that the code used for debugging is consistent with the latest version in the development environment, avoiding inaccurate or erroneous debugging results due to version differences, and reducing repetitive work and communication costs caused by code inconsistencies.

[0148] Figure 3 A flowchart illustrating a debugging method for data development provided in another embodiment of this application is shown below. Figure 3 As shown, the data development debugging method provided in this embodiment is a specific implementation scheme of a distributed operator. The data development debugging method provided in this embodiment includes the following steps:

[0149] S301. In response to receiving a debugging request initiated by the user through the data development platform, the Python operator code to be debugged is synchronized to the interactive debugging environment instance based on the preset synchronization mechanism.

[0150] S302. Get the current cluster resource status.

[0151] S303. Determine the target computing resource group based on the type of Python operator code to be debugged and the current cluster resource status.

[0152] S304. Use a preset container orchestration tool to create corresponding resource objects based on the target computing resource group, and create and start an interactive debugging environment instance in the target computing resource group.

[0153] S305. Execute the Python operator code to be debugged in an interactive debugging environment instance.

[0154] S306. Use a pre-defined distributed tracing framework to obtain the execution path and data flow information of the Python operator code to be debugged.

[0155] S307. Display the execution path and data flow information in the interactive debugging environment instance.

[0156] S308. Backfill the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform.

[0157] S309. In response to the interactive debugging environment instance reaching the preset runtime, release the computing resources corresponding to the interactive debugging environment instance.

[0158] S310. In response to the resource utilization rate of the interactive debugging environment instance being less than the preset utilization rate threshold, release the computing resources corresponding to the interactive debugging environment instance.

[0159] It should be noted that the execution order of S309-S310 is not important.

[0160] In this embodiment, the implementation method and technical effect of S301-S310 are similar to those of the corresponding solutions in the above embodiments, and will not be repeated here.

[0161] Figure 4 A schematic diagram of the debugging device developed for the data provided in this application is shown below. Figure 4 As shown, the data development debugging device 40 provided in this embodiment includes: an allocation and startup module 41, an execution and display module 42, and a backfilling module 43.

[0162] The allocation and startup module 41 is used to respond to a debugging request initiated by a user through the data development platform, allocate computing resources based on the debugging request, and start an interactive debugging environment instance; the debugging request includes the Python operator code to be debugged; the execution and display module 42 is used to execute the Python operator code to be debugged in the interactive debugging environment instance and display the code execution results and variable values; the backfill module 43 is used to backfill the debugged Python operator code in the interactive debugging environment instance into the static code editor of the data development platform.

[0163] The data development debugging device provided in this embodiment can execute... Figure 2 The implementation principles and technical effects of the methods shown are similar, and will not be repeated here.

[0164] Optionally, the allocation and startup module 41, when allocating computing resources and starting an interactive debugging environment instance based on a debugging request, is specifically used to: obtain the current cluster resource status; determine the target computing resource group based on the type of Python operator code to be debugged and the current cluster resource status; create a corresponding resource object based on the target computing resource group using a preset container orchestration tool, and create and start an interactive debugging environment instance in the target computing resource group.

[0165] Optionally, the Python operator code to be debugged is distributed. When the execution and display module 42 executes the Python operator code to be debugged in the interactive debugging environment instance and displays the code execution results and variable values, it is specifically used to: obtain the execution path and data flow information of the Python operator code to be debugged using a preset distributed tracing framework; and display the execution path and data flow information in the interactive debugging environment instance.

[0166] Optionally, the data development debugging device provided in this embodiment also includes a release module.

[0167] Accordingly, the release module is used to release the corresponding computing resources according to preset reclamation rules; the preset reclamation rules include: releasing the computing resources corresponding to the interactive debugging environment instance when the running time of the interactive debugging environment instance reaches a preset running time; and releasing the computing resources corresponding to the interactive debugging environment instance when the resource occupancy rate of the interactive debugging environment instance is less than a preset occupancy rate threshold.

[0168] Optionally, the data development debugging device provided in this embodiment further includes an acquisition module, a generation module, and a synchronization module.

[0169] Accordingly, the acquisition module is used to acquire the environment configuration information of the interactive debugging environment instance and the data development platform; the generation module is used to generate an environment difference report based on the environment configuration information; and the synchronization module is used to synchronize missing configuration items in the environment of the data development platform according to the environment difference report.

[0170] Optionally, the debugging device for data development provided in this embodiment also includes an expansion module.

[0171] Accordingly, the expansion module is used to apply for temporary expansion of memory resources from the target computing resource group if it is determined during code execution that the memory usage rate exceeds a preset usage rate threshold for a first preset duration; the release module is used to release a preset proportion of idle CPU resources if it is determined that the duration of code execution in an idle state reaches a second preset duration.

[0172] Optionally, the synchronization module is also used to synchronize the Python operator code to be debugged to an interactive debugging environment instance based on a preset synchronization mechanism.

[0173] Figure 5 A schematic diagram of the debugging equipment developed for the data provided in this application. (See attached diagram.) Figure 5 As shown, the data development debugging device 50 provided in this embodiment includes a processor 51 and a memory 52. ​​The processor 51 and the memory 52 are connected via a bus and communicate with each other.

[0174] In the specific implementation process, the processor 51 executes the computer execution instructions stored in the memory 52, causing the processor 51 to perform the above-described method.

[0175] The specific implementation process of processor 51 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0176] In the above embodiments, it should be understood that the processor 51 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0177] The memory 52 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0179] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0180] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0181] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0182] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0183] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0186] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0188] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A debugging method for data development, characterized in that, The method includes: In response to receiving a debugging request initiated by a user through the data development platform, computing resources are allocated and an interactive debugging environment instance is launched based on the debugging request; the debugging request includes the Python operator code to be debugged; The Python operator code to be debugged is executed in the interactive debugging environment instance, and the code execution results and variable values ​​are displayed; The debugged Python operator code from the interactive debugging environment instance is then populated back into the static code editor of the data development platform.

2. The method according to claim 1, characterized in that, The process of allocating computing resources and launching an interactive debugging environment instance based on the debugging request includes: Get the current cluster resource status; The target computing resource group is determined based on the type of Python operator code to be debugged and the current cluster resource status; A preset container orchestration tool is used to create corresponding resource objects based on the target computing resource group, and an interactive debugging environment instance is created and started in the target computing resource group.

3. The method according to claim 1, characterized in that, The Python operator code to be debugged is distributed. The execution of the Python operator code in the interactive debugging environment instance, and the display of the code execution results and variable values, include: A pre-defined distributed tracing framework is used to obtain the execution path and data flow information of the Python operator code to be debugged; The execution path and data flow information are displayed in the interactive debugging environment instance.

4. The method according to claim 1, characterized in that, After backfilling the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform, the process further includes: Release the corresponding computing resources according to the preset recycling rules; The preset recycling rules include: In response to the interactive debugging environment instance reaching the preset runtime, the computing resources corresponding to the interactive debugging environment instance are released. In response to the resource utilization rate of the interactive debugging environment instance being less than the preset utilization rate threshold, the computing resources corresponding to the interactive debugging environment instance are released.

5. The method according to claim 1, characterized in that, After backfilling the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform, the process further includes: Obtain the environment configuration information of the interactive debugging environment instance and the data development platform; An environment difference report is generated based on the aforementioned environment configuration information; Based on the environmental difference report, the missing configuration items are synchronized in the environment of the data development platform.

6. The method according to claim 2, characterized in that, Also includes: If it is determined during code execution that the memory usage rate exceeds a preset usage rate threshold for a first preset duration, then a temporary expansion of memory resources is requested from the target computing resource group. If the duration of idle code execution reaches a second preset duration, then a preset proportion of idle CPU resources will be released.

7. The method according to claim 1, characterized in that, Before allocating computing resources based on the debugging request and starting the interactive debugging environment instance, the process also includes: Based on a preset synchronization mechanism, the Python operator code to be debugged is synchronized to the interactive debugging environment instance.

8. A debugging device for data development, characterized in that, include: The allocation and startup module is used to respond to a debugging request initiated by a user through the data development platform, allocate computing resources based on the debugging request, and start an interactive debugging environment instance. The debugging request includes the Python operator code to be debugged; The execution and display module is used to execute the Python operator code to be debugged in the interactive debugging environment instance, and to display the code execution results and variable values; The backfill module is used to backfill the debugged Python operator code from the interactive debugging environment instance into the static code editor of the data development platform.

9. A debugging device for data development, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images