An abnormality detection and repair method and system for a big data task and a medium

Abstract

The present application relates to the field of big data, and provides a big data task exception detection and repair method, system and medium.The big data task exception detection and repair method comprises the following steps: obtaining task information of a big data task with an abnormal task state, wherein the task information comprises a task log; obtaining execution code of the big data task via a decompilation tool, and analyzing the execution code and the task log via an exception analysis module to identify an exception problem; in response to the exception problem being a data source exception and a code exception, analyzing an abnormal code segment, data source information and an exception type via a joint analysis module to determine an exception root cause, wherein the abnormal code segment is identified by the exception analysis module, the data source information is determined by the execution code, and the exception type is determined according to the task log; and triggering abnormal code repair and / or data processing optimization according to the exception root cause.

Description

Technical Field

[0001] This invention relates to the field of big data, and in particular to a method, system, and medium for anomaly detection and repair in big data tasks. Background Technology

[0002] In the digital age, big data technology has penetrated all aspects of various industries, becoming a crucial engine for enterprise development and decision-making. With the continuous increase in data volume and data types, big data tasks are becoming increasingly complex, involving numerous and intricately interconnected technical processes from data collection to analysis and processing. This leads to potential errors and anomalies in big data tasks, which can affect the accuracy and efficiency of task execution, and even cause serious losses to enterprises.

[0003] The execution of existing big data tasks faces numerous challenges and problems. First, because big data tasks typically involve the collaborative work of dozens or even hundreds of servers, it is difficult to quickly and accurately pinpoint the problem when a failure or anomaly occurs, especially in distributed systems. Second, big data tasks often require complex data flows and algorithmic processing. Writing and maintaining this code requires a high level of technical skill, and even highly skilled programmers are prone to errors. Furthermore, when faced with widespread anomalies in big data tasks and the difficulty in accurately locating the cause, multiple factors can lead to the same anomaly. In complex big data processing environments, the root causes of task anomalies may involve multiple aspects, including but not limited to changes in data sources, network communication problems, uneven load on computing nodes, or logical errors in the code itself. The complexity of these intertwined factors renders traditional manual troubleshooting and repair methods inadequate. In addition, the execution environment of big data tasks is usually dynamic, with fluctuations in data volume, changes in network bandwidth, and abnormal data, which can lead to unpredictable problems during task execution.

[0004] Existing technologies for detecting and repairing failures and anomalies in big data tasks still largely rely on manual intervention. While static code analysis tools can help identify some potential problems, they cannot cover all situations; dynamic testing techniques can simulate the task execution process, but they cannot comprehensively cover all possible scenarios; and while log analysis tools can provide some error information, they still cannot achieve targeted automated repair. Therefore, existing technologies often fail to meet the requirements for efficient and stable operation of big data tasks.

[0005] In order to overcome the above-mentioned shortcomings of existing technologies, there is an urgent need in this field for an anomaly detection and repair technology for big data tasks, which can intelligently and automatically realize more accurate and efficient error location and repair strategies, reduce the human cost of locating and investigating anomalies, and better cope with the increasingly complex challenges of big data processing. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0007] To overcome the aforementioned deficiencies in existing technologies, this invention provides a method, system, and medium for anomaly detection and repair in big data tasks. This method can intelligently and automatically achieve more accurate and efficient error location and repair strategies, reduce the manpower cost of locating and troubleshooting anomalies, and better cope with the increasingly complex challenges of big data processing.

[0008] Specifically, the above-mentioned method for anomaly detection and repair of big data tasks according to the first aspect of the present invention includes the following steps: obtaining task information of a big data task whose task status is abnormal, the task information including task logs; obtaining the execution code of the big data task via a decompilation tool, and analyzing the execution code and the task logs via an anomaly analysis module to identify anomalies; in response to the anomalies being data source anomalies and code anomalies, analyzing the abnormal code fragments, data source information and anomaly types via a joint analysis module to determine the root cause of the anomaly, the abnormal code fragments being identified by the anomaly analysis module, the data source information being determined by the execution code, and the anomaly type being determined based on the task logs; and triggering abnormal code repair and / or data processing optimization based on the root cause of the anomaly.

[0009] Furthermore, in some embodiments of the present invention, the anomaly analysis module and the joint analysis module are implemented using the Transformer model, and the joint analysis module has a dual encoder structure.

[0010] Furthermore, in some embodiments of the present invention, the step of analyzing the abnormal code fragment, data source information, and abnormal type via the joint analysis module to determine the root cause of the abnormality includes: analyzing the abnormal code fragment through a code encoder to generate a code semantic vector; analyzing the data source information through a data feature encoder to generate a data feature vector; concatenating and fusing the code semantic vector and the data feature vector to obtain a joint vector; and analyzing the joint vector to determine the probability distribution of the root cause of the abnormality, the root cause of the abnormality, and the confidence level of the root cause of the abnormality.

[0011] Furthermore, in some embodiments of the present invention, the step of analyzing the data source information through a data feature encoder to generate a data feature vector includes: determining the method by which the data feature encoder analyzes the data source information based on the anomaly type.

[0012] Furthermore, in some embodiments of the present invention, the step of determining the method by which the data feature encoder analyzes the data source information based on the anomaly type includes: when the anomaly type is data skew, stack overflow, or memory overflow, extracting data feature vectors by statistically sampling and analyzing the data distribution of the data source information, wherein the data distribution includes field distribution, null value ratio, data size, and number of files; and / or when the anomaly type is exceeding the subscript value, inconsistent data type, or field misalignment leading to null pointer, checking and analyzing the integrity, consistency, and accuracy of the data.

[0013] Furthermore, in some embodiments of the present invention, the anomaly detection and repair method further includes the steps of: in response to the anomaly being a configuration anomaly, collecting and analyzing relevant anomaly information to generate a report and resolution suggestions; and / or in response to the anomaly being a code anomaly, triggering anomaly code repair.

[0014] Furthermore, in some embodiments of the present invention, the steps of abnormal code repair and / or data processing optimization include: refactoring the code based on the original code logic and structure to obtain optimized code, wherein the code refactoring includes code formatting, logical consistency checking and functional integrity verification; and creating and deploying a new deployment package based on the optimized code.

[0015] Furthermore, in some embodiments of the present invention, the anomaly detection and repair method further includes the steps of: re-executing the big data task to obtain the execution data and execution results of the big data task; and storing the execution data and the execution results to a training dataset, and continuously learning and optimizing the anomaly analysis module and the joint analysis module based on the training dataset.

[0016] Furthermore, the anomaly detection and repair system for the aforementioned big data task provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the anomaly detection and repair method for the big data task provided in any of the above embodiments.

[0017] Furthermore, the computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, they implement the anomaly detection and repair method for big data tasks provided in any of the above embodiments.

[0018] The present invention provides a method for anomaly detection and repair in big data tasks, a system for anomaly detection and repair in big data tasks, and a computer-readable storage medium, which have at least the following beneficial effects: The present invention can reduce manual intervention, significantly improve the efficiency of developers, significantly reduce the processing time for resolving task anomalies, and can monitor and acquire anomaly big data tasks in real time and process anomaly tasks in a short time. By automatically repairing code, the present invention reduces the time developers spend manually modifying code and simplifies the development process. Through decompilation and the Transformer model, the present invention scans and analyzes the executable code, identifies outdated and inefficient code, repairs and optimizes it, repackages it after modification, and automatically re-executes it, making the entire process fully automated. Therefore, the present invention can not only accurately locate and repair anomalies in tasks, but also significantly improve the execution efficiency and running efficiency of tasks through intelligent code optimization, maximize resource utilization, complete data processing tasks at the fastest speed, improve the stability and execution efficiency of big data tasks, reduce the risk of task failure, and reduce the amount of anomaly repair and maintenance that developers need to do, thereby allowing more time to develop other new requirements and tasks. Attached Figure Description

[0019] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0020] Figure 1 A schematic diagram of an anomaly detection and repair system for big data tasks provided according to some embodiments of the present invention is shown;

[0021] Figure 2 The diagram illustrates application scenarios provided by some embodiments of the present invention;

[0022] Figure 3A flowchart of an anomaly detection and repair method for big data tasks provided according to some embodiments of the present invention is shown;

[0023] Figure 4 An architecture diagram of a Transformer model provided according to some embodiments of the present invention is shown;

[0024] Figure 5 A flowchart of joint analysis provided according to some embodiments of the present invention is shown.

[0025] Figure label:

[0026] 100: Anomaly detection and repair system for big data tasks;

[0027] 110: Memory;

[0028] 111: Computer-readable storage medium;

[0029] 120: Processor;

[0030] 200: Anomaly Detection and Repair System;

[0031] 300: Hadoop Big Data Cluster;

[0032] S1~S4: Steps. Detailed Implementation

[0033] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention will be presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0034] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0036] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0037] As mentioned above, existing technologies for detecting and repairing failures and anomalies in big data tasks still largely rely on manual intervention. While static code analysis tools can help identify some potential problems, they cannot cover all situations; dynamic testing techniques can simulate the task execution process, but they cannot comprehensively cover all possible scenarios; and while log analysis tools can provide some error information, they still cannot achieve targeted automated repair. Therefore, existing technologies often fail to meet the requirements for efficient and stable operation of big data tasks.

[0038] To overcome the aforementioned deficiencies in existing technologies, this invention provides a method, system, and medium for anomaly detection and repair in big data tasks. This method can intelligently and automatically achieve more accurate and efficient error location and repair strategies, reduce the manpower cost of locating and troubleshooting anomalies, and better cope with the increasingly complex challenges of big data processing.

[0039] In some non-limiting embodiments, the anomaly detection and repair method for the big data task provided in the first aspect of the present invention can be implemented via the anomaly detection and repair system for the big data task provided in the second aspect of the present invention.

[0040] Please refer to Figure 1 , Figure 1 A schematic diagram of an anomaly detection and repair system for big data tasks provided according to some embodiments of the present invention is shown.

[0041] like Figure 1As shown, the anomaly detection and repair system 100 for this big data task may be configured with a memory 110 and a processor 120. The memory 110 includes, but is not limited to, the computer-readable storage medium 111 described in the third aspect of the present invention, which stores computer instructions thereon. The processor 120 is connected to the memory 110 and is configured to execute the computer instructions stored in the memory 110 to implement the anomaly detection and repair method for the big data task provided in the first aspect of the present invention.

[0042] The following will combine Figures 2 to 5 This describes the working principle of the anomaly detection and repair system for the aforementioned big data tasks, as well as the anomaly detection and repair methods for big data tasks executed by the system.

[0043] Those skilled in the art will understand that the embodiments of the anomaly detection and repair method for big data tasks described below are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions that are easy for the public to implement, rather than being intended to limit all functions or all working methods of the anomaly detection and repair system for big data tasks. Similarly, the anomaly detection and repair system for big data tasks described below is also merely a non-limiting implementation provided by the present invention, and does not limit the executing entity and execution order of each step in the anomaly detection and repair method for big data tasks.

[0044] Please refer to Figure 2 and Figure 3 , Figure 2 The diagram illustrates application scenarios provided by some embodiments of the present invention. Figure 3 A flowchart of an anomaly detection and repair method for big data tasks provided according to some embodiments of the present invention is shown.

[0045] like Figure 2 As shown, the anomaly detection and repair system 200 can connect to an external, dynamically running Hadoop big data cluster 300 to monitor big data tasks in the Hadoop big data cluster 300.

[0046] The Hadoop Big Data Cluster 300 is implemented based on Hadoop technology. Hadoop is an open-source distributed computing platform that provides a framework for processing large-scale datasets and supporting large-scale computing tasks. The Hadoop Distributed File System (HDFS) can store large datasets on ordinary hardware and process them in a distributed manner. HDFS divides data into blocks and stores them on multiple nodes in the cluster, and ensures data reliability through a replication mechanism.

[0047] like Figure 3 As shown, the anomaly detection and repair system 200 can perform step S1: obtain the task information of big data tasks whose task status is abnormal, including task logs.

[0048] exist Figure 2 In the illustrated embodiment, the anomaly detection and repair system 200 can utilize the Hadoop ResourceManager REST API interface and multi-threading to obtain real-time task information for all project tasks within the Hadoop big data cluster 300, thereby monitoring all completed tasks within the Hadoop big data cluster 300. Each project task can include different types of big data tasks such as MapReduce tasks, Spark jobs, Spark Streaming jobs, Hive queries, and Flink real-time tasks. The real-time collected task information for various big data tasks can include detailed information such as task submission time, start time, end time, execution node, task logs, and final task status. Real-time collection of task information can improve the processing efficiency of anomaly tasks by the anomaly detection and repair system 200.

[0049] When a task is in an abnormal state, the anomaly detection and repair system 200 can acquire the big data task and store it in a database (such as MySQL).

[0050] In some embodiments, an abnormal task status may be that the final status of the acquired big data task is failed (returning Failed), or the execution time indicated by the start and end times of the big data task exceeds a preset threshold. Alternatively, the anomaly detection and repair system 200 can continuously monitor multiple related or similar big data tasks, and when an indicator indicated by a certain task information exceeds a preset threshold, it can be determined that the big data task is abnormal.

[0051] Preferably, the anomaly detection and repair system 200 can use a final state of failure as a condition for an abnormal task state. When the final state of the task is failure, the anomaly detection and repair system 200 can determine that the big data task has failed abnormally. Figure 2 In the illustrated embodiment, the anomaly detection and repair system 200 can store all big data tasks whose final task status is failure in a database for subsequent analysis and processing. Preferably, the database can mark whether the stored abnormal big data tasks have been processed by setting a status field, so as to avoid multiple threads repeatedly processing the same abnormal data.

[0052] When a status field set in the database indicates that an anomaly in a big data task has not been handled, the anomaly detection and repair system 200 can perform anomaly analysis based on the acquired task information data corresponding to the big data task. In an optional embodiment, the anomaly detection and repair system 200 can acquire the task information of the big data task again and record this task information in the database again when it detects that an anomaly in the big data task has not been handled.

[0053] Task information may include, but is not limited to, execution parameters, task startup classpath, log redirection location, and task logs.

[0054] In some preferred embodiments, the anomaly detection and repair system 200 can use the Transformer model to analyze the execution parameters to determine whether the failure of the big data task is due to the use of excessive or insufficient resources or the lack of necessary execution parameters, and to determine whether there is room for optimization in the big data task.

[0055] The task startup classpath is task information data that indicates the starting location of the execution code for each big data task within a project task. In other words, the task startup classpath can include the starting locations of the execution code for multiple big data tasks within the project task. The starting location of the execution code is typically the startup class code of the big data task that has the function of starting that big data task.

[0056] The anomaly detection and repair system 200 can locate the execution code of big data tasks with abnormal task states based on the task startup classpath for subsequent scanning and analysis by the anomaly analysis module. Specifically, the anomaly detection and repair system 200 can locate the execution code of big data tasks, analyze the configuration files submitted by big data tasks, identify the fully qualified name of the big data task startup class code, and the path of the JAR file based on the task startup classpath.

[0057] The anomaly detection and repair system 200 can use log redirection location to locate the task log, ensuring that the anomaly detection and repair system 200 can easily access and analyze the task log file.

[0058] The task log can record key error information and exception stack traces that led to the failure of big data tasks. The anomaly detection and repair system 200 can extract key error information and mark exception stack traces that may have caused the task failure by analyzing the task log text and exception stack trace information, thereby capturing anomaly patterns.

[0059] In a Hadoop big data cluster 300, the failure of big data task execution may be caused by errors in code logic.

[0060] like Figure 3As shown, the anomaly detection and repair system 200 can perform step S2: obtain the execution code of the big data task through a decompilation tool, and analyze the execution code and task logs through the anomaly analysis module to identify anomalies.

[0061] like Figure 2 As shown, the anomaly detection and repair system 200 is equipped with a trained Transformer model. This Transformer model integrates data from various components, versions, and programming languages ​​in the big data system. The trained Transformer model can effectively handle various types of big data tasks.

[0062] In order to analyze the causes of anomalies in big data tasks in depth, the anomaly detection and repair system 200 can locate the starting position of the execution code through the task startup classpath and obtain the execution code of the big data task through decompilation tools.

[0063] When a big data task experiences an abnormal task status, its execution process typically does not run all of the big data task's code. Instead, it starts from the beginning of the code and terminates at the point where the abnormality occurred.

[0064] Therefore, the above execution code consists of all the code from the starting task code of the big data task to the exception task code.

[0065] The anomaly detection and repair system 200 can perform static code scanning and code logic analysis based on the executable code. Figure 2 In the illustrated embodiment, the anomaly detection and repair system 200 can perform static analysis of the executable code through an anomaly analysis module. The anomaly analysis module can be implemented using a Transformer model.

[0066] Furthermore, the anomaly detection and repair system 200 can also obtain the exception stack information from the task log. The anomaly analysis module then analyzes the task log text and exception stack information to extract key error information and mark exception stacks that may cause task failure.

[0067] In this way, the anomaly analysis module analyzes the execution code and task logs, combining the analysis results of the execution code and the task logs. Based on the attention weight output by the anomaly analysis module, it identifies potential abnormal code segments in the execution code and analyzes the data processing logic within the execution code. This allows it to identify anomalies such as code logic errors or improper data processing that may cause big data task failures. Furthermore, by scanning and analyzing the execution code segments of big data tasks through the anomaly analysis module, the anomaly detection and repair system 200 can gain a comprehensive understanding of the logical flow of big data task execution and potential performance bottlenecks.

[0068] Preferably, the anomaly analysis module can be implemented using a Transformer model with a self-attention mechanism. The Transformer model is a machine algorithm model with a self-attention mechanism that can capture long dependencies and complex logic in code. An anomaly analysis module implemented using a Transformer model has powerful representation capabilities, can handle large-scale code data, and can generate tasks such as code repair and optimization suggestions.

[0069] Please refer to Figure 4 , Figure 4 An architecture diagram of the Transformer model provided according to some embodiments of the present invention is shown.

[0070] like Figure 4 As shown, the Transformer model can have self-attention mechanisms, such as the multi-head attention mechanism and the masked multi-head attention mechanism shown in the figure.

[0071] In big data processing systems, the reasons for task execution failures can be quite complex. In some embodiments, the anomaly identified by the anomaly analysis module may include the following: configuration-related problems caused by configuration anomalies, general code anomalies caused by obvious code data processing logic anomalies, or complex anomalies that include both code anomalies and data source anomalies.

[0072] In some embodiments, when the anomaly is a configuration-related issue caused by a configuration error—that is, when the anomaly analysis module identifies the anomaly as a cluster environment problem, project dependency problem, or other configuration-related issue—the anomaly analysis module can further examine the cluster environment and project dependency configurations. Specifically, the anomaly analysis module can analyze the configuration file of the big data task to check for configuration errors or unreasonable parameter settings; and / or analyze the libraries and versions that the big data task depends on to check for dependency conflicts or version incompatibility issues.

[0073] Configuration issues caused by misconfigurations cannot be resolved through code optimization. Therefore, considering the complexity and global nature of configuration anomalies related to cluster environments and project dependencies, the anomaly detection and repair system does not perform automated repairs to avoid impacting the server environment. Instead, the anomaly analysis module automatically collects and analyzes relevant anomaly information after checking the cluster environment and project dependency configurations. This information is then sent to developers for analysis, ensuring the accuracy of problem resolution and the stability of the server environment.

[0074] In some embodiments, when the anomaly analysis module, after analyzing the data processing logic of the code, finds that the anomaly is a common code anomaly caused by an obvious anomaly in the code's data processing logic, the anomaly detection and repair system 200 can trigger anomaly code repair. Here, the anomaly detection and repair system can store common code anomalies in a database (e.g., MySQL) for the system to identify.

[0075] The anomaly detection and repair system 200 can refactor the code based on the original code logic and structure to obtain optimized code after repair, ensuring that the output results before and after repair remain consistent. Preferably, the anomaly code repair can be implemented by a trained Transformer model. More preferably, the anomaly detection and repair system 200 can save the process data of each repair execution and match similar historical anomaly solutions based on the analysis results of the anomaly analysis module to improve repair efficiency.

[0076] Please continue to refer to this. Figure 3 The anomaly detection and repair system 200 can perform step S3: In response to the anomaly problem being a data source anomaly or a code anomaly, the system analyzes the anomaly code fragment, data source information, and anomaly type via the joint analysis module to determine the root cause of the anomaly.

[0077] When the anomaly identified by the anomaly analysis module is a complex anomaly involving both code and data source anomalies, the situation is often difficult to analyze and resolve using conventional methods. The anomaly detection and repair system 200 needs to conduct further in-depth analysis in conjunction with data source information to improve anomaly handling efficiency and code optimization effectiveness.

[0078] As mentioned above, abnormal code snippets can be identified using an exception analysis module.

[0079] Data source information can be obtained by executing code. The anomaly detection and repair system 200 can scan the executed code to read the data source path and thus collect the relevant data source information. Data source information includes, but is not limited to, data source path, data format, and data volume.

[0080] The exception type can be obtained based on the exception stack information in the task log. In big data processing, the task log usually provides relatively specific exception type information.

[0081] Then, the anomaly detection and repair system 200 can use the joint analysis module to analyze the abnormal code fragments, data source information and anomaly types to further narrow down the scope of anomaly occurrence.

[0082] The joint analysis module can be a Transformer model with a dual-encoder structure. Through the joint analysis module, the anomaly detection and repair system 200 can perform joint code and data source analysis to locate the root cause of the anomaly when the anomaly is related to the data source.

[0083] Please refer to Figure 5 , Figure 5 A flowchart of joint analysis provided according to some embodiments of the present invention is shown.

[0084] like Figure 5 As shown, when the anomaly detection and repair system 200 obtains anomaly task information and analyzes it through the anomaly analysis module to find that the anomaly problem is a complex anomaly problem including data source anomalies and code anomalies, it inputs the code fragments, data source information and anomaly type obtained by the decompilation tool into the joint analysis module.

[0085] The joint analysis module can analyze abnormal code fragments through a code encoder to generate code semantic vectors, and analyze collected data source information through a data feature encoder to generate data feature vectors.

[0086] Here, the anomaly detection and repair system 200 can determine the method of analyzing data source information based on the anomaly type.

[0087] exist Figure 5 In the illustrated embodiment, when the exception stack information in the task log indicates an exception type of data skew, stack overflow, or memory overflow, the joint analysis module can extract data feature vectors by statistically sampling and analyzing the data distribution of the data source information. Data distribution can include field distribution, null value ratio, data size, and number of files. For example, the joint analysis module can analyze the distribution of data size to detect abnormal data size changes, excessively large data sizes, or small data sizes with a high proportion of small files.

[0088] In some embodiments, when the exception type is an out-of-index value, inconsistent data type, or field misalignment leading to a null pointer, the federated analysis module can analyze the data source-related exceptions that cause the big data task to fail by checking and analyzing the integrity, consistency, and accuracy of the data and / or checking the data format to extract data feature vectors.

[0089] After obtaining the code semantic vector and data feature vector, the joint analysis module can concatenate and fuse them to obtain a joint vector. Next, the joint analysis module can analyze the joint vector to achieve root cause diagnosis, determining the probability distribution of abnormal root causes, the abnormal root causes themselves, and the confidence level of the abnormal root causes. Figure 5In the illustrated embodiment, the joint analysis module can perform joint analysis through a fully connected network. The root causes and probability distribution of the anomalies output by the joint analysis module can be: (data skew: 0.8, field misalignment: 0.1, type mismatch: 0.1).

[0090] Please combine Figure 3 and Figure 5 The anomaly detection and repair system 200 can execute step S4: triggering anomaly code repair and / or data processing optimization based on the anomaly root cause.

[0091] The root cause of the anomaly may include code exceptions, and the anomaly detection and repair system 200 can trigger the repair of the exception code. As mentioned above, the anomaly detection and repair system 200 can refactor the code based on the original code logic and structure to obtain optimized code after repair, ensuring that the output results before and after repair remain consistent. Preferably, the exception code repair can be implemented by a trained Transformer model.

[0092] The root cause of the anomaly may include data source anomalies, and the anomaly detection and repair system 200 can trigger data processing optimization. Based on the anomaly type of the data source anomaly, the joint analysis module can further identify whether there are redundant or inefficient operations in the data processing process of the executing code. The joint analysis module can analyze the code and data source anomalies corresponding to the data processing process to determine targeted optimization strategies, such as reducing unnecessary calculations, optimizing algorithms, and improving data structures, thereby improving the execution efficiency and robustness of the code. Preferably, data processing optimization can be implemented by a trained Transformer model.

[0093] Error code repair and / or data processing optimization can both be achieved through code refactoring based on the original code logic and structure using a trained Transformer model. Code refactoring includes code formatting, logical consistency checks, and functional integrity verification. After refactoring, the error detection and repair system 200 can repackage the acquired optimized code into an executable file or component, generating and deploying a new deployment package.

[0094] During deployment, the anomaly detection and repair system 200 will not replace the existing project deployment package, but will create a new deployment package. This effectively avoids the situation where developers have to upload and overwrite the original project deployment package from their local machines after it has been replaced, which would render the previous repair process ineffective.

[0095] In some embodiments, the anomaly detection and repair system 200 re-executes the big data task after repairing or optimizing the anomaly.

[0096] The anomaly detection and repair system 200 can monitor the execution status of big data tasks while re-executing them, including execution time, resource usage, and task logs. Through real-time monitoring of big data tasks, the anomaly detection and repair system 200 can quickly capture and record any possible anomalies or warning messages.

[0097] After the big data task is re-executed, the anomaly detection and repair system 200 can obtain the execution data and results of the big data task. The anomaly detection and repair system 200 can compare the task output results before and after optimization to ensure the effectiveness of the repair and optimization.

[0098] The anomaly detection and repair system 200 can store the acquired execution data and results in a training dataset. Based on this training dataset, the Transformer model in the anomaly detection and repair system 200 is continuously learned and optimized. This Transformer model can be used to implement the anomaly analysis module and the joint analysis module, and it can also be used to repair and optimize abnormal code. Through optimization, improvement, and updating of the Transformer model in the anomaly detection and repair system 200, the accuracy of anomaly detection and the provision of solutions to anomaly problems can be improved, and the code processing logic can be gradually refined and optimized.

[0099] Preferably, such as Figure 2 As shown, the anomaly detection and repair system 200 can also add new anomaly processing data to the training dataset through external network resources, so that the Transformer model in the anomaly detection and repair system 200 can continuously improve its processing capabilities and accuracy.

[0100] In some embodiments, the anomaly detection and repair system 200 can send a detailed report of the successful execution of a big data task to developers via email or other means. The report may include: a description of the anomaly, solutions and optimization measures, details of code changes, task execution results, and performance comparisons. Upon receiving the email, developers can synchronize their local codebase for further manual verification and adjustments. If the adjustments are confirmed, the anomaly detection and repair system 200 can synchronize the optimized code to the local codebase. More preferably, the anomaly detection and repair system 200 can provide detailed synchronization steps and precautions in the report, ensuring that developers can accurately reproduce the optimization process in their local environment, thereby improving their ability to resolve anomalies in big data tasks.

[0101] In some embodiments provided by this invention, the anomaly detection and repair system 200 analyzes task logs to discover that a MapReduce task is experiencing prolonged operation or even failure due to data skew, and monitoring reveals that the MapReduce task status is "Failed". The anomaly detection and repair system 200 automatically captures the task logs of the MapReduce task and the exception stack information within the task logs.

[0102] Using the anomaly analysis module of the Transformer model, the anomaly detection and repair system 200 analyzes the task logs and the execution code obtained by the decompilation tool, identifying the anomalies as data source anomalies and code anomalies. Then, after analyzing the task logs, the anomaly detection and repair system 200 discovered the keyword "Data skew detected in reducer" (data skew was detected in the reducer). Furthermore, the system 200 analyzes the execution code of the MapReduce task obtained by the decompilation tool and obtains the data source information through this code. The anomaly code snippet, data source information, and anomaly type (data skew) are input into the Transformer model's joint analysis module to locate the root cause of the anomaly. It is identified that in the execution code of the MapReduce task, the code snippet related to the GroupBy operation lacked random prefix processing, requiring anomaly code repair and data processing optimization.

[0103] Based on the aforementioned anomalies, the anomaly detection and repair system 200 automatically generates repair code. Specifically, using the Transformer model, the anomaly detection and repair system 200 adds a random prefix to the key before the Shuffle operation in the code snippets related to the GroupBy operation, and then removes the prefix and merges the results at the Reduce end, thereby achieving anomaly code repair and data processing optimization.

[0104] Next, the anomaly detection and repair system 200 repackages, deploys, and executes the repaired task. The execution time after repair is reduced from 3 hours to 40 minutes, resource utilization is reduced by 60%, and the task is successfully completed.

[0105] In summary, the anomaly detection and repair method for big data tasks provided by this invention reduces manual intervention, significantly improves developer efficiency, and drastically reduces the processing time for resolving task anomalies. Furthermore, this invention can monitor and acquire anomaly-prone big data tasks in real time and process them quickly. By automatically repairing code, this invention reduces the time developers spend manually modifying code, simplifying the development process. Through decompilation and the Transformer model, this invention scans and analyzes the executable code, identifying outdated and inefficient code for repair and optimization. After modification, the code is repackaged and automatically re-executed, making the entire process fully automated. Therefore, this invention not only accurately locates and repairs anomalies in tasks but also significantly improves task execution and runtime efficiency through intelligent code optimization, maximizing resource utilization and completing data processing tasks at the fastest speed. This enhances the stability and execution efficiency of big data tasks, reduces the risk of task failure, and reduces the amount of time developers need to spend repairing and maintaining task anomalies, allowing more time to develop other new requirements and tasks.

[0106] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0107] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and skills. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0108] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0109] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0110] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0111] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0112] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for anomaly detection and repair in big data tasks, characterized in that, Including the following steps: Obtain task information for big data tasks with an abnormal task status, including the task startup classpath and task logs; Based on the task startup classpath, the execution code of the big data task is located. The execution code of the big data task is obtained through a decompilation tool. The execution code and the task log are analyzed by an anomaly analysis module to identify anomalies. The anomaly analysis module is implemented by the Transformer model. In response to the anomaly being either a data source anomaly or a code anomaly, the joint analysis module analyzes the anomaly code fragment, data source information, and anomaly type to determine the root cause of the anomaly. The anomaly code fragment is identified by the anomaly analysis module, the data source information is determined by the executed code, and the anomaly type is determined based on the task log. The joint analysis module is implemented using the Transformer model and has a dual encoder structure. The step of analyzing the abnormal code fragment, data source information, and abnormal type via the joint analysis module to determine the root cause of the abnormality includes: analyzing the abnormal code fragment using a code encoder to generate a code semantic vector; analyzing the data source information using a data feature encoder to generate a data feature vector; concatenating and fusing the code semantic vector and the data feature vector to obtain a joint vector; and analyzing the joint vector to determine the probability distribution of the root cause, the root cause of the abnormality, and the confidence level of the root cause of the abnormality; and Based on the root cause of the anomaly, trigger exception code repair and / or data processing optimization.

2. The anomaly detection and repair method for big data tasks as described in claim 1, characterized in that, The step of analyzing the data source information through a data feature encoder to generate a data feature vector includes: The method by which the data feature encoder analyzes the data source information is determined based on the anomaly type.

3. The anomaly detection and repair method for big data tasks as described in claim 2, characterized in that, The steps for determining the method by which the data feature encoder analyzes the data source information based on the anomaly type include: When the anomaly type is data skew, stack overflow, or memory overflow, statistical sampling analysis is performed on the data distribution of the data source information to extract a data feature vector. The data distribution includes field distribution, null value ratio, data size, and number of files; and / or When the exception type is an out-of-index value, inconsistent data type, or field misalignment leading to a null pointer, check and analyze the integrity, consistency, and accuracy of the data.

4. The anomaly detection and repair method for big data tasks as described in claim 1, characterized in that, The anomaly detection and repair method also includes the following steps: In response to the aforementioned anomaly being a configuration anomaly, relevant anomaly information is collected and analyzed to generate a report and resolution recommendations; and / or In response to the aforementioned exception being a regular code exception stored in the database, an exception code repair is triggered.

5. The anomaly detection and repair method for big data tasks as described in claim 4, characterized in that, The steps for abnormal code repair and / or data processing optimization include: Code refactoring is performed based on the existing code logic and structure to obtain optimized code. This code refactoring includes code formatting, logical consistency checks, and functional integrity verification. A new deployment package is created and deployed based on the optimized code.

6. The anomaly detection and repair method for big data tasks as described in claim 1, characterized in that, The anomaly detection and repair method also includes the following steps: Re-execute the big data task to obtain the execution data and results of the big data task; and The execution data and execution results are stored in a training dataset, and the anomaly analysis module and the joint analysis module are continuously learned and optimized based on the training dataset.

7. An anomaly detection and repair system for big data tasks, characterized in that, include: Memory, on which computer instructions are stored; as well as A processor, connected to the memory, is configured to execute computer instructions stored in the memory to implement the anomaly detection and repair method for big data tasks as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the anomaly detection and repair method for big data tasks as described in any one of claims 1 to 6 is implemented.

Images