Data pipeline exception processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, can be applied to the field of science and technology finance / digital medical treatment, and discloses a data pipeline exception processing method and device, electronic equipment and a storage medium. The method comprises the steps of monitoring a task execution state in a data pipeline, and collecting abnormal log information and abnormal context information when execution abnormity occurs; performing semantic analysis on the exception log by using a pre-trained large model, determining an exception reason type and generating an exception processing strategy; executing automatic repair or strategy execution operation according to the exception handling strategy; the task scheduling priority, the execution sequence and the resource allocation are optimized in combination with the task running state and the resource use state in the exception processing process, and the overall processing efficiency is improved; and performing associated storage on the exception log, the exception reason, the processing strategy and the processing result, and continuously updating the exception processing strategy library. According to the method, intelligent diagnosis, self-repairing and continuous stability optimization of data pipeline abnormity are realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology and can be applied to the fields of financial technology / digital healthcare, and in particular to a data pipeline anomaly handling method, device, electronic device and storage medium. Background Technology

[0002] As information systems expand in scale, data pipelines, as a crucial data processing mechanism for multi-source data acquisition, cleaning, transformation, verification, and output, are widely used in business scenarios such as fintech, insurance, risk control, regulatory reporting, and digital healthcare. Traditional data pipeline systems are typically built on task scheduling frameworks, executing data processing tasks through pre-configured task dependencies and scheduling strategies. When anomalies occur during task execution, the system generally marks the task as failed based on process return status or interface response results, recording corresponding execution logs and alarm information, which are then manually analyzed by operations personnel based on the log content. In traditional technologies, anomaly identification mainly relies on error code matching or simple rule judgments, resulting in anomaly information scattered across multiple log records, making it difficult to form a complete description of the cause and correlation of anomalies at the system level. For example, when a payment system experiences delays, traditional methods rely solely on error codes, failing to identify the root cause of network or interface failures in a timely manner, potentially leading to transaction delays and increased operational risks. When medical equipment disconnects from the information system, traditional systems recover through simple retries, failing to address equipment malfunctions or network problems, thus affecting patient treatment progress. In terms of exception handling, traditional systems often employ fixed-number retries or manual reruns. When exceptions originate from external interfaces, network environments, or changes in resource status, repeatedly executing the same task path may still produce the same failure result. The exception handling process is independent of the scheduling system's execution logic. For example, a payment interface failure might be recovered through retries, but this fails to resolve the underlying issue, leading to repeated errors and impacting transaction processing. Similarly, repeated medical equipment failures, where retries do not address the root cause, affect high-priority tasks such as emergency care. Regarding scheduling strategies, task priorities and resource allocation are typically statically configured, making dynamic adjustments based on task failures and operational status difficult during scheduling. In fintech scenarios with high timeliness requirements, such as accounting and report generation, task delays can easily affect data delivery progress. For instance, task delays may affect report generation and transaction clearing, increasing financial risk; emergency tasks or patient treatment priorities cannot be adjusted in real time, leading to delays. Furthermore, exception handling experience is often scattered and manually recorded, lacking a systematic connection between exception logs, processing results, and subsequent task scheduling, resulting in similar exceptions recurring in different tasks. Summary of the Invention

[0003] The main technical problem addressed by the embodiments of this application is the low efficiency of traditional data pipelines in terms of exception handling and task scheduling.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a data pipeline anomaly handling method, comprising: monitoring the execution status of tasks in a target data pipeline; when a task execution anomaly is detected, executing an error event capture process and collecting anomaly log information and anomaly context information corresponding to the task execution anomaly; inputting the anomaly log information and the anomaly context information into a pre-trained large model diagnostic engine, performing semantic parsing on the anomaly log information, determining the anomaly cause type based on the parsing result, and generating an anomaly handling strategy corresponding to the anomaly cause type; performing automated repair operations or strategy execution operations to handle the task execution anomaly according to the anomaly handling strategy; during the anomaly handling process, optimizing the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline based on the task running status and resource usage status; associating and storing the anomaly log information, the anomaly cause type, the anomaly handling strategy, and the processing result, and updating a preset anomaly handling strategy library based on the association storage result.

[0005] Optionally, the step of monitoring the execution status of tasks in the target data pipeline, and when a task execution anomaly is detected, executing an error event capture process and collecting the anomaly log information and anomaly context information corresponding to the task execution anomaly, includes: capturing a task execution failure event through the task scheduler; generating a trigger signal for a task execution anomaly when the task execution return code is not equal to a preset success code or the interface response status is a preset anomaly status; collecting execution log information, system resource status information, and task dependency context information corresponding to the failed task when capturing the task execution failure event; and constructing an error event data structure to characterize the task execution anomaly based on the collected execution log information, system resource status information, and task dependency context information.

[0006] Optionally, the step of inputting the abnormal log information and the abnormal context information to a pre-trained large model diagnostic engine to perform semantic parsing on the abnormal log information includes: calling the pre-trained large model diagnostic engine to parse the abnormal log information according to a preset semantic parsing template, converting the unstructured log text into a structured log semantic representation; when performing semantic parsing on the abnormal log information, semantically associating the abnormal description information, error type information, and execution environment information in the log text according to the abnormal context information; and outputting structured log semantic data to characterize task execution abnormalities based on the semantic parsing results, wherein the structured log semantic data includes at least an abnormal category identifier and an abnormal description field.

[0007] Optionally, the step of determining the anomaly cause type based on the parsing results and generating an anomaly handling strategy corresponding to the anomaly cause type includes: processing the semantic parsing results of the anomaly log information and classifying task execution anomalies to determine the anomaly cause type corresponding to the task execution anomaly; in the process of determining the anomaly cause type, by accessing a knowledge base constructed from historical anomaly handling data, retrieving historical anomaly records that match the current anomaly log information and the anomaly context information to obtain historical reference information associated with the anomaly cause type; after obtaining the historical reference information, comparing the current anomaly log information, the anomaly context information, and the historical reference information to determine the direct cause information and potential root cause information of the task execution anomaly; generating an anomaly handling strategy corresponding to the task execution anomaly according to the analysis results of the anomaly cause type, the direct cause information, and the potential root cause information, the anomaly handling strategy including operation instructions or strategy suggestion data for task anomaly handling; when generating the anomaly handling strategy, sequentially processing and verifying the anomaly cause type, the direct cause information, and the potential root cause information according to a preset causal analysis process.

[0008] Optionally, the step of performing automated repair operations or strategy execution operations to handle task execution anomalies according to the anomaly handling strategy includes: parsing the anomaly handling strategy to determine whether the anomaly handling strategy corresponds to a preset standardized processing operation; when the anomaly handling strategy corresponds to a standardized processing operation, calling the interface or control instruction corresponding to the standardized processing operation to trigger the automated repair operation or strategy execution operation; when the anomaly handling strategy corresponds to a non-standardized processing operation, generating a task to be executed corresponding to the anomaly handling strategy and submitting the task to be executed to the approval process; before executing the automated repair operation or strategy execution operation, performing strategy verification on the anomaly handling strategy to confirm that the anomaly handling strategy meets preset execution conditions; and after the anomaly handling strategy verification passes, executing the automated repair operation or strategy execution operation corresponding to the anomaly handling strategy.

[0009] Optionally, the step of optimizing the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline based on task running status and resource usage status during the exception handling process includes: obtaining task type information, task running status information, exception occurrence frequency information, and resource usage status information of each task in the target data pipeline; optimizing the scheduling priority of tasks in the target data pipeline based on the task type information, task running status information, and exception occurrence frequency information; optimizing the parallelism parameters or task allocation nodes of tasks in the target data pipeline when the resource usage status is detected to reach a preset resource usage threshold condition; optimizing the execution order of tasks and updating the scheduling order relationship of tasks based on the dependencies between tasks in the target data pipeline; and adding the corresponding task back to the scheduling queue after the exception handling is completed to continue executing the tasks in the target data pipeline.

[0010] Optionally, the step of associating and storing the exception log information, the exception cause type, the exception handling strategy, and the handling result, and updating the preset exception handling strategy library based on the association storage result, includes: storing the exception log information, the exception cause type, the exception handling strategy, and the handling result corresponding to each task execution exception in an exception handling history library to obtain historical exception handling records; performing feature processing on the historical exception handling records and writing the processed historical exception handling records into a semantic index database; when a new task execution exception occurs, retrieving a target historical exception handling record from the semantic index database that matches the current exception log information and exception context information; writing the retrieved target historical exception handling record as reference information into the exception handling strategy library; and updating the exception handling strategy in the exception handling strategy library based on the exception handling record and the handling result.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a data pipeline anomaly handling device, comprising: a task execution status monitoring module, used to monitor the execution status of tasks in a target data pipeline; when a task execution anomaly is detected, executing an error event capture process and collecting anomaly log information and anomaly context information corresponding to the task execution anomaly; a large model diagnostic engine module, used to input the anomaly log information and the anomaly context information into a pre-trained large model diagnostic engine, perform semantic parsing on the anomaly log information, determine the anomaly cause type based on the parsing result, and generate an anomaly handling strategy corresponding to the anomaly cause type; an anomaly handling strategy execution module, used to execute automated repair operations or strategy execution operations according to the anomaly handling strategy to handle the task execution anomaly; a data pipeline task optimization module, used to optimize the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline according to the task running status and resource usage status during the anomaly handling process; and an anomaly handling strategy update module, used to associate and store the anomaly log information, the anomaly cause type, the anomaly handling strategy, and the processing result, and update a preset anomaly handling strategy library according to the associated storage result.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data pipeline exception handling method as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a computer-readable storage medium that stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the data pipeline exception handling method as described above.

[0014] Unlike related technologies, this application introduces an anomaly semantic understanding and causal analysis mechanism centered on a large model. This transforms the identification, diagnosis, and handling of task execution anomalies in the data pipeline from a traditional rule-based or manual experience-dependent approach to an automated processing method based on semantic parsing and reasoning decision-making. Anomaly logs are no longer merely used for post-incident investigation but directly participate in anomaly cause determination and handling strategy generation. Simultaneously, a mechanism combining automated repair and controlled execution is introduced during anomaly handling. Standardized anomalies can be handled quickly and autonomously, while high-risk anomalies are executed in an orderly manner under compliance constraints, thus balancing processing efficiency and system security. Furthermore, by dynamically optimizing scheduling strategies in conjunction with task running status and resource usage status during the anomaly repair phase, the impact of anomaly handling on the overall operation of the data pipeline is effectively controlled. Continuous accumulation and feedback updates of anomaly logs, cause types, handling strategies, and processing results ensure that anomaly handling capabilities evolve with system operation. In summary, this application achieves closed-loop processing of data pipeline anomalies from discovery, understanding, decision-making to execution and learning, significantly improving the intelligence level of anomaly handling, execution consistency, and the stability and adaptability of the data pipeline under complex operating environments. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the data pipeline anomaly handling method provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the data pipeline exception handling method provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of semantic parsing in the data pipeline exception handling method provided in this application embodiment.

[0019] Figure 4 This is a schematic diagram of the execution flow for handling task execution exceptions in the data pipeline exception handling method provided in this application embodiment.

[0020] Figure 5 This is a schematic diagram of the system structure of the data pipeline anomaly handling device provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of the electronic device for executing the data pipeline exception handling method provided in the embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0023] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0024] It should be noted that the user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0026] To facilitate understanding of this embodiment, a detailed description of a data pipeline anomaly handling method disclosed in this application embodiment will be provided first. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the data pipeline anomaly handling method provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the data pipeline exception handling method provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this data pipeline exception handling method can be implemented by the processor calling computer-readable instructions stored in the memory. Figure 1The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0027] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the data pipeline exception handling method provided in the embodiments of this application, such as... Figure 2 As shown, it includes the following steps: S1. Monitor the execution status of tasks in the target data pipeline. When an abnormal task execution is detected, execute the error event capture process and collect the abnormal log information and abnormal context information corresponding to the abnormal task execution.

[0028] By unifying the perception of task execution results at the task scheduling level, a set of runtime information related to task execution can be promptly solidified when anomalies occur. By centrally aggregating task execution logs, system resource status information, and task dependency information, a complete and consistent description of the anomaly event can be formed within the time window of the anomaly, avoiding the dispersion of anomaly information across different components or log systems. This approach provides a stable data foundation and clear contextual boundaries for subsequent anomaly semantic parsing, anomaly cause identification, and processing strategy generation.

[0029] In fintech data processing scenarios, tasks such as transaction clearing, risk indicator calculation, and account reconciliation are typically scheduled and executed uniformly by a task scheduler. When a transaction processing task experiences an abnormal process exit or an external service call fails during execution, the task scheduler can identify the anomaly by detecting a task failure signal where the task execution return code is not equal to the preset success code or the interface response status is HTTP 5xx. Around this task failure signal, the system synchronously collects task execution logs, system resource status information such as CPU and I / O, and dependency context content such as upstream transaction data input status and configuration version information. It then aggregates the exception stack information, error type information, and execution context information according to a preset log template, thereby forming a complete error event record at the first moment of financial data processing task failure. This provides an accurate data foundation for subsequent anomaly cause analysis and processing strategy generation.

[0030] In digital healthcare data processing scenarios, tasks such as patient data processing, medical device monitoring, and electronic medical record (EMR) systems are typically scheduled and executed uniformly by a task scheduler. When a medical processing task experiences an abnormal process exit or device connection failure during execution, the task scheduler can identify the anomaly by detecting a task failure signal where the task execution return code is not equal to the preset success code or the interface response status is HTTP 5xx. Around this task failure signal, the system synchronously collects contextual information such as task execution logs, medical device status (e.g., connection status of ECG monitors or imaging equipment), network connectivity, patient data (e.g., real-time monitored physiological indicators), and upstream medical system data (e.g., laboratory results or drug information). Following a preset log template, the system aggregates anomaly stack information, error type information, and execution environment information, thereby forming a complete error event record at the first moment of medical task failure. This provides an accurate data foundation for subsequent anomaly cause analysis and processing strategy generation.

[0031] As an optional implementation, the process of monitoring the execution status of tasks in the data pipeline in step S1 above may also specifically include the following steps S11 to S13.

[0032] S11. Capture task execution failure events through the task scheduler. When the task execution return code is not equal to the preset success code or the interface response status is the preset abnormal status, generate a task execution exception trigger signal.

[0033] By unifying the perception of task execution results at the task scheduling level, anomaly detection logic can be directly bound to the task execution result without relying on additional anomaly recognition models or complex rule-based judgment mechanisms. By using task execution status as the basis for anomaly triggering, anomaly signals can be generated instantaneously when task execution fails, thereby shortening the anomaly detection chain and reducing misjudgments and system overhead caused by log scanning or status inference.

[0034] S12. When capturing a task execution failure event, collect the execution log information, system resource status information, and task dependency context information corresponding to the failed task.

[0035] Once a task execution failure is identified, a complete snapshot of the system state at the time of the anomaly can be preserved by synchronously collecting relevant information about the task's runtime environment. By simultaneously acquiring task execution logs, system resource runtime status, and task dependency information, issues such as missing information or inconsistent timing during anomaly analysis can be avoided, thus providing a comprehensive and consistent data foundation for subsequent anomaly root cause analysis.

[0036] S13. Based on the collected execution log information, system resource status information, and task dependency context information, construct an error event data structure to characterize task execution anomalies.

[0037] By uniformly encapsulating anomaly-related information from multiple sources, scattered log content, resource status information, and dependency context information can be integrated into a structured expression of anomaly events. Building a standardized error event data structure provides a unified data interface for subsequent anomaly semantic parsing, cause type identification, and processing strategy generation, while reducing the coupling complexity between different modules.

[0038] Through steps S11 to S13, the abnormal event is quickly perceived and uniformly modeled at the first moment when the task execution anomaly occurs. This allows the anomaly judgment process to be directly based on the task execution result, thereby reducing the anomaly identification delay and the probability of misjudgment. At the same time, by structurally encapsulating the log information, resource status information, and dependency context information at the time of the anomaly, a complete and consistent description of the anomaly event can be formed. This provides a standardized data input foundation for subsequent anomaly semantic parsing, cause type identification, and processing strategy generation, thereby improving the overall stability and scalability of the data pipeline anomaly handling process.

[0039] S2. Input the abnormal log information and abnormal context information into the pre-trained large model diagnostic engine, perform semantic parsing on the abnormal log information, determine the abnormal cause type based on the parsing results, and generate an abnormal handling strategy corresponding to the abnormal cause type.

[0040] By introducing a pre-trained large language model to process anomaly logs, traditional rule- or keyword-matching-based log analysis methods can be overcome, transforming unstructured log content into structured diagnostic information with clear semantic meaning. Combining task runtime environment information and historical anomaly handling knowledge, comprehensive reasoning can be performed on the error patterns, dependencies, and causal clues hidden in the anomaly logs, thereby achieving hierarchical identification of anomaly cause types, direct causes, and potential root causes. Simultaneously, by standardizing the expression of diagnostic results, the anomaly analysis output has a consistent data structure, providing a stable and reusable input foundation for subsequent anomaly handling strategy generation and automated execution.

[0041] In fintech data processing scenarios, core business processes such as transaction clearing, real-time risk control, and fund monitoring typically rely on multiple data pipelines running in parallel, frequently calling external payment interfaces, account services, and risk control engines. When a risk control calculation task encounters connection timeout logs and the number of retries reaches a preset limit, the large-scale model's error diagnosis and repair suggestion module can perform semantic analysis on abnormal log fragments and environmental information, transforming unstructured error information into structured diagnostic results. It clarifies the anomaly as a network timeout, directly pointing to an upstream interface request failure, and further identifies potential root causes as domain name resolution anomalies. Based on this, by retrieving repair experience related to interface timeouts and network failures from the historical anomaly handling knowledge base, the large-scale model can generate actionable repair suggestions, such as switching to a backup payment interface or delaying retry execution. This provides decision support with business semantic understanding and engineering feasibility for automated anomaly handling of financial data pipelines.

[0042] In the field of digital healthcare, when anomalies occur in medical data processing tasks, the collected anomaly log information and anomaly context information are input into a pre-trained large-scale model diagnostic engine for semantic parsing. This log information may include device malfunction reports, abnormal patient data input, and errors returned by medical software interfaces, while the context information may include device type, patient medical record information, and network connection status. The large-scale model diagnostic engine analyzes this unstructured log text, converting it into structured log semantic data and identifying the type of anomaly (e.g., device malfunction, network disconnection, data format error). Based on the parsing results, the diagnostic engine determines the root cause of the anomaly (e.g., device connection timeout, sensor malfunction, data transmission interruption) and generates corresponding anomaly handling strategies. For example, if the cause is a device connection timeout, the system may suggest restarting the device or switching to a backup device; if it's a data format error, it may prompt for data verification or format conversion; if it's a network disconnection, the system can automatically attempt to reconnect or switch to a backup network path. These handling strategies can directly drive automatic repair operations, reducing manual intervention and improving the response speed and stability of the medical data processing system.

[0043] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram of the execution flow of semantic parsing in the data pipeline exception handling method provided in this application embodiment, such as... Figure 3 As shown, it may include the following steps S21 to S23.

[0044] S21. Based on the preset semantic parsing template, the pre-trained large model diagnostic engine is called to parse and process the abnormal log information, converting the unstructured log text into a structured log semantic representation.

[0045] By introducing pre-trained large language models into log processing, the log analysis process, which originally relied on human experience or rule matching, can be transformed into an automated process based on semantic understanding. Using a unified semantic parsing template, the model's output format can be constrained, ensuring that key information in abnormal logs is expressed according to a fixed semantic structure. This transforms scattered, unstructured text logs into structured semantic representations that are easier for computational systems to process further.

[0046] S22. When performing semantic parsing on the exception log information, semantic association is performed on the exception description information, error type information and execution environment information in the log text based on the exception context information.

[0047] Incorporating runtime environment information into log semantic parsing allows anomaly analysis to move beyond a single log entry and instead incorporate comprehensive judgments based on task execution dependencies, configuration status, and runtime conditions. By semantically associating anomaly descriptions and error types with execution environment information within the log text, the contextual completeness of anomaly semantic representation can be enhanced, reducing the risk of misjudgments due to insufficient information in log fragments.

[0048] S23. Based on the semantic parsing results, output structured log semantic data to characterize task execution anomalies. The structured log semantic data includes at least an anomaly category identifier and an anomaly description field.

[0049] By uniformly outputting the semantic parsing results, a stable data representation can be formed to characterize the core features of task execution anomalies. By explicitly including anomaly category identifiers and anomaly description fields in the structured log semantic data, a standardized input interface can be provided for subsequent anomaly cause type determination, processing strategy generation, and automated repair execution, thereby reducing the complexity of data integration between different processing modules.

[0050] As an example, in a data pipeline anomaly handling scenario, for the task identified as task_32, when the anomaly log fragment shows "Connection Error: timeout to host …", and the runtime environment information shows that the dependent interface is API_A and the number of retries is 3, the large model diagnostic engine can process the log content according to the preset log semantic parsing template, transforming the unstructured log information into a structured causal chain expression, and outputting the cause category as network timeout, the direct cause as upstream API_A request failure, and the potential root cause as DNS resolution anomaly. At the same time, it provides repair suggestions such as switching to the backup interface API_B or delaying retries, forming a standardized diagnostic result that includes cause type, cause details, confidence level, and suggested measures. On this basis, by retrieving task configurations, upstream dependencies, and repair experience information related to interface timeout and dependency anomalies from the historical knowledge base, the diagnostic result is subjected to context enhancement processing, thereby obtaining anomaly diagnostic output that integrates historical experience, providing a more complete semantic basis for the generation and execution of subsequent anomaly handling strategies.

[0051] As another optional implementation, the process of generating the exception handling strategy in step S2 above may also include the following steps S24 to S28.

[0052] S24. Process the semantic parsing results of the abnormal log information, classify the task execution abnormalities, and determine the abnormal cause type corresponding to the task execution abnormality.

[0053] By further processing the structured semantic results output from the log semantic parsing stage, task execution anomalies can be transitioned from the text understanding level to the anomaly category determination level. By classifying the anomaly manifestations, complex and diverse anomalies can be mapped to a unified anomaly cause type, thereby providing a clear classification entry point for subsequent anomaly handling logic and avoiding branching expansion in the anomaly handling process due to differences in anomaly descriptions.

[0054] S25. In the process of determining the type of anomaly cause, by accessing the knowledge base built from historical anomaly handling data, historical anomaly records that match the current anomaly log information and anomaly context information are retrieved to obtain historical reference information associated with the anomaly cause type.

[0055] Introducing historical anomaly handling knowledge into the anomaly cause type determination process allows for comparison of the current anomaly within the semantic space of existing anomaly handling experience. By accessing a knowledge base constructed from historical anomaly handling data and performing similarity searches, historical anomaly records that are similar to the current anomaly's manifestation and operating environment characteristics can be obtained, thus providing a referenceable empirical information basis for anomaly cause determination.

[0056] S26. After obtaining historical reference information, compare and process the current exception log information, exception context information and historical reference information to determine the direct cause information and potential root cause information of the task execution exception.

[0057] By comparing current anomaly-related information with retrieved historical reference information, we can identify the causal differences and common characteristics behind the anomalies. This comparative analysis method can distinguish between superficial anomalies and deeper causes, thereby clarifying the direct triggers and potential systemic root causes of the anomalies, providing a more targeted basis for formulating subsequent handling strategies.

[0058] S27. Based on the analysis results of the anomaly cause type, direct cause information and potential root cause information, generate an anomaly handling strategy corresponding to the task execution anomaly. The anomaly handling strategy includes operation instructions or strategy suggestion data for handling task anomalies.

[0059] After identifying the type of anomaly, its direct cause, and potential root causes, the anomaly diagnosis results can be transformed into actionable solutions. By mapping anomaly diagnosis information to operational instructions or strategy recommendations, the anomaly handling process can seamlessly transition from the analysis phase to the execution phase, providing clear action descriptions for automated repair or controlled execution.

[0060] S28. When generating an anomaly handling strategy, the anomaly cause type, the direct cause information, and the potential root cause information are processed sequentially and their consistency is verified according to the preset causal analysis process.

[0061] Introducing a pre-defined causal analysis process during the anomaly handling strategy generation can unify the processing order of anomaly cause information at different levels. By verifying the consistency of anomaly cause types, direct cause information, and potential root cause information, causal conflicts or logical inconsistencies in the handling strategy can be avoided, thereby improving the stability and controllability of the anomaly handling strategy during the execution phase.

[0062] As an example, during the anomaly diagnosis process, the large model performs anomaly attribution and strategy generation processing according to a preset multi-level analysis instruction chain. The analysis instruction chain includes at least anomaly type identification instructions, root cause analysis instructions, and repair strategy generation instructions. The analysis instructions at each level are triggered sequentially according to the causal relationship, which is used to constrain the reasoning process of the large model layer by layer, ensuring that the anomaly diagnosis results maintain a consistent causal logical relationship between anomaly type, direct cause, potential root cause, and repair strategy.

[0063] Through steps S21 to S28, the pre-trained large model's capabilities in semantic understanding, causal reasoning, and knowledge integration are fully utilized. This allows the anomaly handling process to move beyond rule matching or simple classification judgments, instead achieving anomaly diagnosis based on the model's comprehensive reasoning of anomaly log semantics, runtime context, and historical experience. Through semantic structured parsing, anomaly cause type determination, and identification of direct and potential root causes led by the large model, complex anomaly scenarios can be mapped into diagnostic results with clear causal relationships. Simultaneously, the large model's reasoning capabilities in context enhancement and multi-level causal analysis instruction chains enable the generation of processing strategies that match the anomaly causes, resulting in higher consistency, interpretability, and executability of anomaly handling results. This enhances the overall intelligence and engineering applicability of the data pipeline anomaly handling process.

[0064] S3. Based on the exception handling strategy, perform automated repair operations or strategy execution operations to handle task execution exceptions.

[0065] Once the cause of the anomaly has been identified and an anomaly handling strategy has been formulated, extending the anomaly handling process from the analysis phase to the execution phase can achieve an automated closed-loop anomaly handling process. By parsing the anomaly handling strategy and differentiating handling actions based on different risk levels, the system can automatically complete controllable repair operations while ensuring security and compliance. Simultaneously, a controlled execution mechanism is introduced for handling actions with potential impact. By introducing a strategy verification process before execution, it can be ensured that anomaly handling operations comply with established operating rules and control strategies, thereby reducing the impact of the anomaly repair process on data pipeline stability and business continuity.

[0066] In the data processing scenarios of fintech systems, data pipeline tasks such as transaction clearing, risk control calculation, and account reconciliation often rely on external interfaces, authentication services, and upstream business systems. When the large-scale model diagnostic results indicate that the anomaly is caused by network timeout and the repair suggestion is to switch to a backup interface, the system can directly call the interface management service to route transaction requests to the backup payment API and continue executing data processing tasks. When the diagnostic results indicate permission anomalies and the repair suggestion is to refresh authentication information, the system can automatically trigger the OAuth credential refresh process to restore access to account services or risk control services. When the anomaly involves data logic errors or inconsistent upstream dependencies, the system can generate a rollback task and submit it to the approval process. After confirmation by operations and maintenance, the task version rollback or data source mirroring switch operation will be performed. Through the above methods, after anomalies occur in financial data processing, the repair suggestions generated by the large-scale model can be transformed into repair actions with compliance verification and execution control capabilities, thereby ensuring the continuity and security of financial business processing flows.

[0067] In the field of digital healthcare, once the specific causes and solutions for anomalies are identified based on an anomaly handling strategy generated from a large model, the system executes automated repair operations according to that strategy. For example, if a diagnostic result indicates that a medical device (such as a monitor) failed to execute a task due to a network connection timeout, the system will automatically trigger a retry operation or switch to a backup network path. If the anomaly is caused by a hardware failure (such as sensor malfunction), the system may automatically activate a backup device and switch the data stream to ensure uninterrupted data acquisition and transmission. In some cases, if the anomaly handling strategy involves high-risk operations or requires manual confirmation (such as data rollback or device restart), the system will generate a task pending approval for maintenance personnel. For more complex or high-risk operations (such as replacing critical medical equipment or restoring patient historical data), the system ensures that all repair operations are performed within a compliant security framework and conducts strategy verification to ensure compliance with medical data privacy and security requirements. Through this combination of automated repair and strategy execution, medical data processing systems can quickly resume normal operation, reduce manual intervention, and improve the continuity and efficiency of the diagnosis and treatment process.

[0068] As an alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram of the execution flow for handling task execution exceptions in the data pipeline exception handling method provided in this application embodiment, as shown below. Figure 4 As shown, it may include the following steps S31 to S35.

[0069] S31. Analyze the exception handling strategy and determine whether the exception handling strategy corresponds to the preset standardized processing operation.

[0070] By performing structured analysis of the exception handling strategy, the processing suggestions generated by the large model can be mapped to processing types that the system can recognize. By distinguishing whether there is processing content that matches the preset operation template, different execution paths can be established during the exception handling phase, thus providing a clear basis for determining whether subsequent automatic or controlled execution will occur.

[0071] S32. When the exception handling strategy corresponds to the standardized handling operation, call the interface or control instruction corresponding to the standardized handling operation to trigger the automated repair operation or strategy execution operation.

[0072] When an exception handling strategy matches a predefined standard processing operation in the system, the corresponding repair action can be triggered via interface calls or control commands. By directly converting exception handling strategies into system-level execution commands, the exception handling response chain can be shortened, enabling rapid and automated handling of common exception scenarios.

[0073] S33. When the exception handling strategy corresponds to a non-standardized processing operation, generate a task to be executed corresponding to the exception handling strategy and submit the task to be executed to the approval process.

[0074] For anomaly handling strategies not included in standardized processing, risk isolation can be achieved during the anomaly handling process by generating independent tasks to be executed and introducing an approval process. By incorporating high-impact or highly uncertain processing operations into manual confirmation or operation and maintenance review mechanisms, uncontrollable impacts on data pipeline operation caused by anomaly repair actions can be avoided.

[0075] S34. Before performing automated repair operations or policy execution operations, perform policy verification on the exception handling policy to confirm that the exception handling policy meets the preset execution conditions.

[0076] Before exception handling operations enter the execution phase, a policy verification mechanism can be introduced to uniformly check the legality of the handling policy and the execution conditions. By verifying the operating rules, permission scope, and system status, it can be ensured that exception handling operations comply with established control requirements, thereby improving the security and stability of the exception handling process.

[0077] S35. After the exception handling strategy is verified, execute the automatic repair operation or strategy execution operation corresponding to the exception handling strategy.

[0078] Once the strategy verification results meet the preset execution conditions, the exception handling strategy can be transformed into actual execution actions. By triggering repair or execution operations only after verification is completed, a closed loop for exception handling can be completed while ensuring compliance, enabling the exception handling process to have controllable execution and stable implementation capabilities.

[0079] Through steps S31 to S35, the anomaly handling strategies generated by the large model can be transformed into controllable and executable system-level processing actions, enabling a smooth transition from the strategy decision-making stage to the execution stage in the data pipeline anomaly handling process. By determining the type of the anomaly handling strategies and distinguishing between standardized and non-standardized processing operations, automated repair of common anomalies can be achieved while ensuring system security, and a controlled execution mechanism can be introduced for high-risk processing behaviors. By introducing a unified strategy verification process before execution, it can be ensured that the anomaly handling operations comply with established operating rules and compliance requirements, thereby reducing the potential impact of the anomaly repair process on the stability of financial data processing and business continuity, and improving the reliability and engineering feasibility of the data pipeline anomaly handling process.

[0080] S4. During the exception handling process, optimize the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline based on the task running status and resource usage status.

[0081] Introducing a dynamic scheduling optimization mechanism during anomaly handling can maintain the overall operational efficiency and resource utilization stability of the data pipeline during the anomaly repair and recovery phases. By comprehensively considering the task running status, anomaly characteristics, and system resource usage, dynamic decisions can be made regarding task scheduling order, execution priority, and resource allocation methods, thereby preventing the anomaly handling process from having a cascading impact on other normal tasks. Furthermore, by incorporating anomaly diagnosis results into the scheduling decision logic, scheduling behavior can acquire business awareness, enabling critical and high-risk tasks to receive more reasonable execution arrangements during the anomaly recovery phase.

[0082] In fintech scenarios, tasks such as real-time risk control and fund clearing typically run within the same data pipeline scheduling system, with varying requirements for timeliness and stability. When some risk control analysis tasks need to be re-executed due to anomaly repair, and system resource utilization approaches a preset threshold, the dynamic scheduling optimization module can recalculate scheduling priorities based on anomaly frequency, task type, and resource utilization, giving higher execution weights to regulatory or fund-related tasks. Simultaneously, when high computing node load is detected, the scheduling module can automatically reduce the parallelism of some non-critical tasks or migrate repaired tasks to backup computing nodes for continued execution. In scenarios with task dependencies, the scheduling module can also dynamically adjust task dependency paths based on anomaly diagnosis results output by the large model, prioritizing the continuous execution of clearing links and critical risk control paths, thereby maintaining the overall stability and business continuity of the financial data processing flow during the anomaly recovery phase.

[0083] In the field of digital healthcare, when a task anomaly is detected and processing begins, the scheduling priority, execution order, or resource allocation parameters of the task are dynamically optimized based on its operational status and resource usage. For example, when an emergency task occurs, its priority is automatically increased to ensure that emergency-related data (such as real-time monitoring data and vital signs) can be transmitted and processed preferentially. If equipment malfunctions or data streams are delayed, the system will adjust the execution order of other tasks or allocate more resources to critical tasks based on the current health status and resource utilization of the equipment to ensure that the patient's real-time data is processed in a timely manner. For instance, if a hospital's imaging equipment (such as a CT scanner) is processing multiple tasks, and the system detects that one task cannot be completed due to data transmission problems, the system will automatically adjust based on equipment resource usage and task importance. This may involve reassigning the task to other available equipment or prioritizing the equipment for more urgent tasks. If the equipment load is too high, the system may also dynamically adjust the parallelism of tasks and allocate them to other backup equipment to avoid task delays caused by resource constraints. In addition, if the system detects insufficient network bandwidth, it may adjust the data transmission frequency of non-urgent tasks or move data transmission tasks to periods when the network is not in use, in order to ensure that critical medical data (such as real-time monitoring data and emergency data) can be transmitted and processed at the best time, thus ensuring the timeliness and accuracy of medical decisions.

[0084] As an optional implementation, the optimization process of tasks in the data pipeline in step S4 above may also include the following steps S41 to S45.

[0085] S41. Obtain task type information, task running status information, anomaly frequency information, and resource usage status information for each task in the target data pipeline.

[0086] By uniformly acquiring task types, running status, frequency of anomalies, and resource usage status, a set of status information reflecting the overall operational status of the data pipeline can be formed. This allows subsequent scheduling and resource decisions to be based on complete and real-time operational data, avoiding scheduling decisions based solely on a single indicator.

[0087] S42. Optimize the scheduling priority of tasks in the target data pipeline based on task type information, task running status information, and anomaly occurrence frequency information.

[0088] By optimizing scheduling priorities based on comprehensive task business attributes, current operating status, and anomaly occurrence characteristics, the data pipeline can prioritize the execution order of critical and high-risk tasks during the anomaly handling phase, thereby reducing the impact of anomalies on core business processes and improving the overall execution efficiency during the anomaly recovery phase.

[0089] S43. When the resource usage status is detected to reach the preset resource usage threshold, optimize the parallelism parameter of the task or the task allocation node in the target data pipeline.

[0090] Optimizing the parallelism parameters or task allocation nodes when resource usage reaches a preset threshold can alleviate the problem of scarce computing resources or unbalanced load, avoid resource contention between abnormal repair tasks and normal tasks, and thus improve the operational stability of the data pipeline under high load scenarios.

[0091] S44. Optimize the execution order of tasks based on the dependencies between tasks in the target data pipeline and update the task scheduling order.

[0092] By optimizing the execution order based on the dependencies between tasks, it can be ensured that critical dependency paths are scheduled for execution first, reducing cascading delays caused by task blocking or unreasonable ordering, thereby improving the overall throughput of the data pipeline during the anomaly recovery process.

[0093] S45. After the exception handling is completed, the corresponding task is added back to the scheduling queue to continue executing the tasks in the target data pipeline.

[0094] After the exception handling is completed, the relevant tasks are put back into the scheduling queue, which can achieve a seamless connection between exception repair and normal scheduling process, so that the data pipeline can smoothly transition from the exception handling state to the normal operation state, ensuring the continuity and integrity of data processing tasks.

[0095] Through steps S41 to S45, a comprehensive understanding of the data pipeline's operational status is achieved during the anomaly handling phase. Based on this understanding, task scheduling priorities, resource allocation methods, and execution order are dynamically optimized, ensuring a stable execution rhythm for the data pipeline during anomaly repair. By comprehensively considering resource usage and task dependencies, resource contention and task blocking issues can be alleviated, ensuring that critical and high-risk tasks are completed with priority during the anomaly recovery phase. Simultaneously, by smoothly reintegrating relevant tasks into the scheduling system after anomaly handling, effective integration of the anomaly handling process with the normal scheduling process can be achieved, thereby improving the overall stability, recovery efficiency, and continuous operation capability of the data pipeline in complex operating environments.

[0096] S5. Link and store the exception log information, exception cause type, exception handling strategy and handling result, and update the preset exception handling strategy library according to the linked storage result.

[0097] By associating and storing anomaly log information, anomaly cause types, anomaly handling strategies, and handling results, a complete closed-loop record of anomaly handling can be formed during the data pipeline anomaly handling process. This ensures that the background of each anomaly, diagnostic conclusions, and handling results are traceable and reusable. By continuously updating the anomaly handling strategy library based on the associated storage results, the anomaly handling strategies can evolve with the system operation, avoiding long-term static solidification of strategies. This improves the consistency of the data pipeline's response to recurring and similar anomalies in complex operating environments and enhances the stability of its handling.

[0098] In fintech systems, data pipeline tasks such as transaction clearing, risk assessment, and account reconciliation may repeatedly encounter anomalies such as interface timeouts, permission failures, or unstable external data sources during long-term operation. When an anomaly occurs in a clearing task, the system stores the corresponding anomaly log fragment, anomaly cause tag, the adopted repair strategy, and the final repair effect in an anomaly experience library. The system also performs semantic feature processing on the anomaly event and writes it to a vector index database. Subsequently, when a new financial data processing task presents a situation highly similar to historical anomalies in terms of log semantics and operating environment characteristics, the large model can automatically retrieve similar cases from the anomaly experience library as context input during anomaly diagnosis, thereby generating new processing strategies by referencing previous root cause judgments and repair effects. Simultaneously, by continuously incorporating anomaly events, diagnostic results, and repair feedback into the knowledge update process, the anomaly handling knowledge in financial business scenarios can be gradually enriched, making the anomaly diagnosis results of the large model in subsequent transaction clearing or risk control tasks more aligned with the operational characteristics and business constraints of the financial field.

[0099] In the field of digital healthcare, after anomaly handling is completed, the system stores the anomaly log information, anomaly cause type, anomaly handling strategy, and handling result in a correlated manner. For example, when a task anomaly caused by a device failure is handled, the system stores the relevant anomaly logs (e.g., device disconnection, data loss), anomaly cause type (e.g., hardware failure, communication interruption), the repair strategy adopted (e.g., restarting the device, switching to a backup device), and the handling result (e.g., device recovery, normal data transmission), and correlates them with historical anomaly records. This stored information not only includes the handling details of the current anomaly but also correlates with other related tasks and historical anomalies in the system to facilitate subsequent anomaly analysis and optimization. The stored data is entered into an anomaly experience library and organized using semantic indexes. The system regularly updates the anomaly handling strategy library to ensure that new anomaly handling experience is effectively fed back into the strategy library. For example, if a certain device failure mode occurs frequently, the system may automatically adjust the preset handling strategy, increasing the frequency of preventative maintenance for this type of device or incorporating new repair steps into the standard operating procedure. When similar anomalies occur again, the system can retrieve relevant repair strategies and handling methods from its experience base by matching historical data, providing a reference for diagnosis and repair, quickly reducing manual intervention and improving repair efficiency. This continuously updated mechanism ensures that the medical device management system can continuously improve its processing capabilities and adaptability, especially when dealing with various device and data anomalies in complex medical environments.

[0100] As an optional implementation, the process of updating the exception handling strategy library in step S5 above may also include the following steps S51 to S52.

[0101] S51. Store the exception log information, exception cause type, exception handling strategy and handling result corresponding to each task execution exception in the exception handling history database to obtain the historical exception handling record.

[0102] By fully recording each anomaly handling process, the anomaly background, anomaly diagnosis conclusion, and actual handling results can be formed into a unified data entity, thereby constructing an anomaly historical data foundation with temporal continuity and causal integrity, providing a reliable data source for subsequent anomaly knowledge accumulation and strategy evolution.

[0103] S52. Perform feature processing on the historical anomaly handling records and write the processed historical anomaly handling records into the semantic index database.

[0104] By performing feature processing on historical anomaly handling data, the original records can be transformed into an expression suitable for semantic retrieval and similarity calculation. Writing the processed anomaly records into a semantic index database helps establish semantic relationships between anomaly events, providing efficient data support for subsequent similar anomaly retrieval.

[0105] S53. When a new task execution exception occurs, retrieve the target historical exception handling record from the semantic index database that matches the current exception log information and exception context information.

[0106] When a new anomaly occurs, by performing a similarity search in the semantic index database, historical anomaly records that are highly similar to the current anomaly in terms of semantic features and operating environment can be quickly located. This enables the anomaly diagnosis process to have the ability to refer to historical experience, avoiding the need to start the analysis from scratch for each anomaly handling.

[0107] S54. Write the retrieved target historical anomaly handling records into the anomaly handling strategy library as reference information.

[0108] By introducing the retrieved historical anomaly records into the anomaly handling strategy library, the anomaly handling strategies can be linked with the repair experience verified in actual operation, thereby enriching the content structure of the strategy library and making anomaly handling decisions have experience-driven characteristics.

[0109] S55. Update the exception handling strategies in the exception handling strategy library based on the exception handling records and processing results.

[0110] By feeding back the results of exception handling and applying them to the exception handling strategy library, the exception handling strategies can be continuously evolved. This allows the strategy library to be updated as exception handling practices are implemented, thereby improving the adaptability and stability of exception handling strategies in recurring and similar exception scenarios.

[0111] Through steps S51 to S55, the exception logs, cause determinations, processing strategies, and execution results generated during the data pipeline exception handling process are precipitated into structured exception history knowledge, enabling the exception handling process to have complete traceability and reusability. By performing feature processing on the exception history and establishing a semantic index, it is possible to support the rapid retrieval of similar exception scenarios, so that subsequent exception diagnosis and strategy generation are supported by historical experience. At the same time, by continuously feeding back historical exception records and processing results to the exception handling strategy library, the exception handling strategy can be continuously updated and improved with operational practice, thereby improving the data pipeline's adaptability, processing consistency, and overall operational stability in recurring and complex exception scenarios.

[0112] The data pipeline anomaly handling method provided in this application introduces an anomaly semantic understanding and causal analysis mechanism centered on a large model. This transforms unstructured anomaly logs generated during task execution into diagnostic results with clear semantics and causal relationships. Based on these results, an executable anomaly handling strategy is generated, achieving closed-loop processing of anomaly identification, cause determination, strategy generation, and execution. Simultaneously, by combining anomaly diagnostic results with automated repair, controlled execution, and dynamic scheduling mechanisms, the data pipeline can be rationally scheduled according to task importance, operational status, and resource usage during the anomaly occurrence and recovery phases, reducing the impact of anomalies on overall operational efficiency and business continuity. Furthermore, by associating and storing log information, cause types, processing strategies, and execution results during the anomaly handling process, and accumulating knowledge, the anomaly handling strategy can continuously evolve and feed back into subsequent anomaly diagnoses, thereby improving the intelligence level, adaptability, and long-term operational stability of the data pipeline in complex operating environments.

[0113] Please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the system structure of the data pipeline anomaly handling device provided in the embodiments of this application, such as... Figure 5 As shown, the data pipeline anomaly handling device 50 includes: a task execution status monitoring module 51, a large model diagnostic engine module 52, an anomaly handling strategy execution module 53, a data pipeline task optimization module 54, and an anomaly handling strategy update module 55. The task execution status monitoring module 51 is specifically used to monitor the execution status of tasks in the target data pipeline. When a task execution anomaly is detected, an error event capture process is executed, and anomaly log information and anomaly context information corresponding to the task execution anomaly are collected. The large model diagnostic engine module 51 is specifically used to input the anomaly log information and the anomaly context information into the pre-trained large model diagnostic engine, perform semantic parsing on the anomaly log information, determine the anomaly cause type based on the parsing result, and generate anomaly handling strategy corresponding to the anomaly cause type. The anomaly handling strategy execution module 53 is specifically used to execute automated repair operations or strategy execution operations according to the anomaly handling strategy to handle the task execution anomaly. The data pipeline task optimization module 54 is specifically used to optimize the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline according to the task running status and resource usage status during the anomaly handling process. The anomaly handling strategy update module 55 is specifically used to associate and store the anomaly log information, the anomaly cause type, the anomaly handling strategy, and the processing result, and update the preset anomaly handling strategy library according to the associated storage result.

[0114] As an optional implementation, the task execution status monitoring module 51 is further configured to capture task execution failure events through the task scheduler, and generate a task execution exception trigger signal when the task execution return code is not equal to the preset success code or the interface response status is a preset exception status; when capturing the task execution failure event, it collects the execution log information, system resource status information and task dependency context information corresponding to the failed task; and constructs an error event data structure to characterize the task execution exception based on the collected execution log information, system resource status information and task dependency context information.

[0115] As an optional implementation, the large model diagnostic engine module 52 is further specifically used to call the pre-trained large model diagnostic engine to parse the abnormal log information according to the preset semantic parsing template, converting the unstructured log text into a structured log semantic representation; when performing semantic parsing on the abnormal log information, semantic association is performed on the abnormal description information, error type information and execution environment information in the log text according to the abnormal context information; based on the semantic parsing results, structured log semantic data for characterizing task execution abnormalities is output, wherein the structured log semantic data includes at least an abnormal category identifier and an abnormal description field.

[0116] As an optional implementation, the large model diagnostic engine module 52 is further specifically used to process the semantic parsing results of the abnormal log information, classify task execution abnormalities, and determine the abnormal cause type corresponding to the task execution abnormality; in the process of determining the abnormal cause type, by accessing a knowledge base constructed from historical abnormality handling data, historical abnormal records matching the current abnormal log information and the abnormal context information are retrieved to obtain historical reference information associated with the abnormal cause type; after obtaining the historical reference information, the current abnormal log information, the abnormal context information, and the historical reference information are compared to determine the direct cause information and potential root cause information of the task execution abnormality; according to the analysis results of the abnormal cause type, the direct cause information, and the potential root cause information, an abnormality handling strategy corresponding to the task execution abnormality is generated, the abnormality handling strategy including operation instructions or strategy suggestion data for task abnormality handling; when generating the abnormality handling strategy, the abnormal cause type, the direct cause information, and the potential root cause information are processed sequentially and verified for consistency according to a preset causal analysis process.

[0117] As an optional implementation, the exception handling strategy execution module 53 is further configured to parse the exception handling strategy and determine whether the exception handling strategy corresponds to a preset standardized processing operation; when the exception handling strategy corresponds to a standardized processing operation, it calls the interface or control instruction corresponding to the standardized processing operation to trigger the automated repair operation or strategy execution operation; when the exception handling strategy corresponds to a non-standardized processing operation, it generates a task to be executed corresponding to the exception handling strategy and submits the task to be executed to the approval process; before executing the automated repair operation or strategy execution operation, it performs strategy verification on the exception handling strategy to confirm that the exception handling strategy meets the preset execution conditions; when the exception handling strategy verification passes, it executes the automated repair operation or strategy execution operation corresponding to the exception handling strategy.

[0118] As an optional implementation, the data pipeline task optimization module 54 is further configured to acquire task type information, task running status information, anomaly frequency information, and resource usage status information of each task in the target data pipeline; optimize the scheduling priority of tasks in the target data pipeline based on the task type information, task running status information, and anomaly frequency information; optimize the parallelism parameters or task allocation nodes of tasks in the target data pipeline when the resource usage status is detected to reach a preset resource usage threshold; optimize the execution order of tasks and update the scheduling order relationship of tasks based on the dependencies between tasks in the target data pipeline; and add the corresponding task back to the scheduling queue after the anomaly handling is completed to continue executing the tasks in the target data pipeline.

[0119] As an optional implementation, the exception handling strategy update module 55 is further configured to store the exception log information, exception cause type, exception handling strategy, and processing result corresponding to each task execution exception in an exception handling history database to obtain historical exception handling records; perform feature processing on the historical exception handling records and write the processed historical exception handling records into a semantic index database; when a new task execution exception occurs, retrieve a target historical exception handling record that matches the current exception log information and exception context information from the semantic index database; write the retrieved target historical exception handling record as reference information into the exception handling strategy database; and update the exception handling strategy in the exception handling strategy database according to the exception handling record and the processing result.

[0120] It should be noted that the above-described data pipeline anomaly handling device can execute the data pipeline anomaly handling method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the data pipeline anomaly handling device can be found in the data pipeline anomaly handling method provided in the embodiments of this application.

[0121] Figure 6 This is a schematic diagram of the hardware structure of the electronic device for executing the data pipeline exception handling method provided in the embodiments of this application, as shown below. Figure 6 As shown, the electronic device 600 includes: One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0122] The processor 610 and the memory 620 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0123] The memory 620, as a computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the data pipeline exception handling method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the data pipeline exception handling method in the above-described method embodiments.

[0124] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the data pipeline exception handling device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories may be connected to the data pipeline exception handling device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0125] The one or more modules are stored in the memory 620. When executed by the one or more processors 610, they perform the data pipeline exception handling method in any of the above method embodiments, for example, performing the above-described... Figure 2 Method steps S1 to S5, Figure 3 Method steps S21 to S23, Figure 4Steps S31 to S35 of the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0126] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0127] This application provides a computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 610 can enable the one or more processors to execute the data pipeline exception handling method in any of the above method embodiments, for example, to execute the above-described... Figure 2 Method steps S1 to S5, Figure 3 Method steps S21 to S23, Figure 4 Steps S31 to S35 of the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0128] This application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by an electronic device, enable the electronic device to perform the data pipeline exception handling method described in any of the above method embodiments, for example, to perform the above-described... Figure 2 Method steps S1 to S5, Figure 3 Method steps S21 to S23, Figure 4 Steps S31 to S35 of the method are implemented. Figure 5 The functions of modules 51-55 in the document.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data pipeline anomaly handling method, characterized in that, include: Monitor the execution status of tasks in the target data pipeline. When an abnormal task execution is detected, execute the error event capture process and collect the abnormal log information and abnormal context information corresponding to the abnormal task execution. Input the abnormal log information and the abnormal context information into the pre-trained large model diagnostic engine, perform semantic parsing on the abnormal log information, determine the abnormal cause type based on the parsing result, and generate an abnormal handling strategy corresponding to the abnormal cause type. According to the aforementioned exception handling strategy, perform automated repair operations or strategy execution operations to handle the task execution exception; During the exception handling process, the scheduling priority, execution order, or resource allocation parameters of the tasks in the target data pipeline are optimized based on the task running status and resource usage status. The abnormal log information, the abnormal cause type, the abnormal handling strategy and the handling result are associated and stored, and the preset abnormal handling strategy library is updated according to the associated storage result.

2. The data pipeline anomaly handling method according to claim 1, characterized in that, The monitoring of the execution status of tasks in the target data pipeline, when an execution anomaly is detected, includes the following steps: executing an error event capture process and collecting the anomaly log information and anomaly context information corresponding to the task execution anomaly. The task scheduler captures task execution failure events. When the task execution return code is not equal to the preset success code or the interface response status is a preset exception status, a task execution exception trigger signal is generated. When capturing the task execution failure event, collect the execution log information, system resource status information and the task dependency context information corresponding to the failed task; Based on the collected execution log information, system resource status information, and task dependency context information, an error event data structure is constructed to characterize task execution anomalies.

3. The data pipeline anomaly handling method according to claim 1, characterized in that, The step of inputting the abnormal log information and the abnormal context information into a pre-trained large model diagnostic engine, and performing semantic parsing on the abnormal log information, includes: According to the preset semantic parsing template, the pre-trained large model diagnostic engine is invoked to parse and process the abnormal log information, converting the unstructured log text into a structured log semantic representation. When performing semantic parsing on the abnormal log information, semantic association is performed on the abnormal description information, error type information, and execution environment information in the log text based on the abnormal context information; Based on the semantic parsing results, structured log semantic data is output to characterize task execution anomalies. The structured log semantic data includes at least an anomaly category identifier and an anomaly description field.

4. The data pipeline anomaly handling method according to claim 1, characterized in that, The step of determining the anomaly cause type based on the parsing results and generating an anomaly handling strategy corresponding to the anomaly cause type includes: The semantic parsing results of the abnormal log information are processed, and the task execution abnormalities are classified to determine the abnormal cause type corresponding to the task execution abnormality. In the process of determining the type of anomaly cause, a knowledge base constructed from historical anomaly processing data is accessed to retrieve historical anomaly records that match the current anomaly log information and the anomaly context information, thereby obtaining historical reference information associated with the anomaly cause type. After obtaining the historical reference information, the current abnormal log information, the abnormal context information and the historical reference information are compared and processed to determine the direct cause information and potential root cause information of the task execution abnormality; Based on the analysis results of the abnormal cause type, the direct cause information and the potential root cause information, an abnormality handling strategy corresponding to the task execution abnormality is generated. The abnormality handling strategy includes operation instructions or strategy suggestion data for task abnormality handling. When generating an anomaly handling strategy, the anomaly cause type, the direct cause information, and the potential root cause information are processed sequentially and their consistency is verified according to a preset causal analysis process.

5. The data pipeline anomaly handling method according to claim 1, characterized in that, The step of handling task execution exceptions by performing automated repair operations or strategy execution operations according to the exception handling strategy includes: The exception handling strategy is parsed to determine whether it corresponds to a preset standardized processing operation. When the exception handling strategy corresponds to a standardized processing operation, the interface or control instruction corresponding to the standardized processing operation is called to trigger the automated repair operation or strategy execution operation. When the exception handling strategy corresponds to a non-standardized processing operation, a task to be executed corresponding to the exception handling strategy is generated and the task to be executed is submitted to the approval process. Before executing the automated repair operation or the strategy execution operation, the exception handling strategy is validated to confirm that the exception handling strategy meets the preset execution conditions. Once the exception handling strategy passes the verification, the automated repair operation or strategy execution operation corresponding to the exception handling strategy is executed.

6. The data pipeline anomaly handling method according to claim 1, characterized in that, The step of optimizing the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline based on the task running status and resource usage status during the exception handling process includes: Obtain task type information, task running status information, anomaly frequency information, and resource usage status information for each task in the target data pipeline; Based on the task type information, task running status information, and anomaly occurrence frequency information, the scheduling priority of tasks in the target data pipeline is optimized; When the resource usage status is detected to have reached a preset resource usage threshold, the parallelism parameter of the task or the task allocation node in the target data pipeline is optimized. Based on the dependencies between tasks in the target data pipeline, optimize the execution order of tasks and update the scheduling order of tasks. Once the exception is handled, the corresponding task is added back to the scheduling queue to continue executing the tasks in the target data pipeline.

7. The data pipeline anomaly handling method according to claim 1, characterized in that, The step of associating and storing the exception log information, the exception cause type, the exception handling strategy, and the handling result, and updating the preset exception handling strategy library based on the association storage result, includes: The exception log information, exception cause type, exception handling strategy and handling result corresponding to each task execution exception are stored in the exception handling history database to obtain historical exception handling records. The historical anomaly handling records are characterized and then written into the semantic index database. When a new task execution exception occurs, retrieve the target historical exception handling record that matches the current exception log information and exception context information from the semantic index database; The retrieved historical anomaly handling records of the target are written into the anomaly handling strategy library as reference information. Based on the exception handling records and the processing results, the exception handling strategies in the exception handling strategy library are updated.

8. A data pipeline anomaly handling device, characterized in that, include: The task execution status monitoring module is used to monitor the execution status of tasks in the target data pipeline. When a task execution abnormality is detected, the error event capture process is executed, and the abnormal log information and abnormal context information corresponding to the task execution abnormality are collected. The large model diagnostic engine module is used to input the abnormal log information and the abnormal context information into the pre-trained large model diagnostic engine, perform semantic parsing on the abnormal log information, determine the abnormal cause type based on the parsing result, and generate an abnormal handling strategy corresponding to the abnormal cause type. The exception handling strategy execution module is used to perform automated repair operations or strategy execution operations according to the exception handling strategy to handle the task execution exception; The data pipeline task optimization module is used to optimize the scheduling priority, execution order, or resource allocation parameters of tasks in the target data pipeline based on the task running status and resource usage status during the exception handling process. The exception handling strategy update module is used to associate and store the exception log information, the exception cause type, the exception handling strategy and the processing result, and update the preset exception handling strategy library according to the associated storage result.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by an electronic device, cause the electronic device to perform the method according to any one of claims 1-7.

Images