A task exception processing method and device

Abstract

The application discloses a task exception processing method and device, and relates to the technical field of intelligent exception analysis. The method embodiment can include: obtaining a running log corresponding to a task, and splitting the running log into multiple log blocks; matching a first exception category for the multiple log blocks from multiple exception levels included in a pre-constructed exception tree and at least one exception category included in each exception level; determining an exception reason configured for the first exception category corresponding to the multiple log blocks according to the exception reason pre-configured for each exception category; and providing the first exception category and the exception reason configured for the first exception category to a task manager, so that the task manager repairs the task according to the first exception category and the exception reason configured for the first exception category. The embodiment can intelligently analyze task exceptions through logs, and discover and process abnormal tasks in time.

Description

Technical Field

[0001] This invention relates to the field of intelligent anomaly analysis technology, and in particular to a method and apparatus for handling task anomalies. Background Technology

[0002] Currently, data processing involves the execution of various data processing tasks, and any anomaly in any of these tasks can cause problems in data processing. For example, databases or data warehouses mainly involve data processing tasks such as extraction, transformation, fabrication, and push, as well as data processing chains composed of these tasks. They process data through independent data processing tasks or chains. A problem with any data processing task will lead to data processing anomalies in the database or data warehouse. Therefore, it is necessary to analyze and repair the problematic tasks to ensure the normal operation of the data processing process.

[0003] Currently, the main method for locating problematic tasks and their causes is through manual log analysis. However, manual analysis is time-consuming, resulting in high manpower costs, and the timeliness of task exception handling is poor. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a task anomaly handling method and apparatus, which can intelligently analyze task anomalies through logs, promptly detect and handle abnormal tasks, thereby ensuring the timeliness of task anomaly handling and reducing the cost of manual analysis of task anomalies.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a task exception handling method, comprising:

[0006] Obtain the execution log corresponding to the task, and split the execution log into multiple log blocks;

[0007] A first anomaly category is matched for the log block from multiple anomaly levels included in the pre-constructed anomaly tree and at least one anomaly category included in each anomaly level;

[0008] Based on the exception reasons pre-configured for each exception category, determine the exception reason configured for the first exception category corresponding to the log block;

[0009] The first exception category and the exception reason configured for the first exception category are provided to the task manager so that the task manager can repair the task based on the first exception category and the exception reason configured for the first exception category.

[0010] Optionally, splitting the runtime log into multiple log blocks includes:

[0011] Based on the multiple log recording times included in the operation log, the log text located between every two adjacent log recording times is determined as a log block.

[0012] Optionally, each anomaly category included in the initial anomaly level among the multiple anomaly levels included in the anomaly tree is obtained by clustering historical operation logs;

[0013] For each other anomaly level besides the initial anomaly level, each anomaly category included in the other anomaly level is obtained by clustering the multiple anomaly categories included in the previous anomaly level corresponding to the other anomaly level.

[0014] Optionally, each of the anomaly levels is provided with a corresponding anomaly similarity threshold;

[0015] The step of matching the log block to the first anomaly category includes:

[0016] For each of the log blocks, perform the following operations:

[0017] Obtain the core information for clustering from the log block;

[0018] Calculate the similarity between the core information and the anomaly categories included in the multiple anomaly levels;

[0019] Based on the similarity and the anomaly similarity threshold set by the anomaly level, the first anomaly category matching the log block is selected.

[0020] Optionally, calculating the similarity between the core information and the anomaly categories included in the multiple anomaly levels includes:

[0021] The core information is segmented into words to obtain one or more word groups; and if any of the aforementioned exception levels is the current exception level, the following operations are performed:

[0022] Calculate the maximum string length and the longest substring length of the core information based on one or more of the phrases and the exception categories included in the current exception level;

[0023] Using the maximum length of the string and the length of the longest substring of the core information, the similarity between the core information and the anomaly categories included in the previous anomaly level is calculated.

[0024] Optionally, the filtering of the first anomaly category matching the log block includes:

[0025] The following operations N1 to N3 are performed cyclically for each of the aforementioned exception levels:

[0026] N1: Locate the current anomaly level for the core information from among the multiple anomaly levels;

[0027] N2: In the current anomaly level, search for a second anomaly category, wherein the similarity between the second anomaly category and the core information is greater than or equal to the anomaly similarity threshold corresponding to the current anomaly level;

[0028] N3: Generate a temporary exception category based on the core information of the log block and the second exception category;

[0029] N4: Compare the temporary anomaly category with the second anomaly category; if the comparison result indicates that the temporary anomaly category is consistent with the second anomaly category, then execute N5; if the comparison result indicates that the temporary anomaly category is inconsistent with the second anomaly category, then execute N6.

[0030] N5: Determine that the second exception category is the first exception category that matches the log block, delete the temporary exception category, end the process and end the loop;

[0031] N6: Take the next exception level corresponding to the current exception level as the current exception level and execute operation N1.

[0032] Optionally, the generated temporary exception categories include:

[0033] In the case where the current exception level is the initial exception level, the core information of the log block and the second exception category searched from the initial exception level are aggregated to form a temporary exception category;

[0034] For cases where the current anomaly level is an anomaly level other than the initial anomaly level, a temporary anomaly category corresponding to the previous anomaly level is determined. The temporary anomaly category corresponding to the previous anomaly level and the second anomaly category searched for in the current anomaly level are aggregated to form a temporary anomaly category.

[0035] Optionally, after searching for the second anomaly category and before generating the temporary anomaly category, the method further includes:

[0036] If no second anomaly category is searched, the next anomaly level corresponding to the current anomaly level is directly taken as the current anomaly level, and the step of locating the current anomaly level for the core information from multiple anomaly levels is executed again.

[0037] Optionally, after searching for the second anomaly category, the method further includes:

[0038] If the second anomaly category is not found, a new anomaly category is generated for the core information, and the new anomaly category is added to the current anomaly level.

[0039] Optionally, after comparing the temporary anomaly category with the second anomaly category, the method further includes:

[0040] If the comparison result indicates that the temporary anomaly category is inconsistent with the second anomaly category, the temporary anomaly category is added to the current anomaly level.

[0041] Optionally, before setting the next exception level corresponding to the current exception level as the current exception level, the method further includes:

[0042] If the comparison results indicate that the temporary anomaly category is inconsistent with the second anomaly category.

[0043] Determine whether the current exception level is the highest level of the exception tree. If it is, end the current process; otherwise, execute the step of taking the next exception level corresponding to the current exception level as the current exception level.

[0044] Optionally, generating a new anomaly category for the core information includes:

[0045] The core information is directly identified as a new anomaly category.

[0046] Optionally, generating a new anomaly category for the core information includes:

[0047] For cases where the current exception level is any level other than the initial exception level,

[0048] Determine the previous anomaly level corresponding to the other anomaly levels;

[0049] The temporary exception categories generated at the previous exception level and the original exception categories included at the previous exception level are aggregated to form new exception categories for the other exception levels.

[0050] Optionally, after splitting the runtime log into multiple log blocks and before matching the log blocks with a first exception category, the method further includes:

[0051] Each log block is formatted according to a preset log format.

[0052] The process of obtaining core information for clustering from the log block includes:

[0053] Based on the core characteristics of the pre-defined log blocks, extract core information from the formatted log blocks.

[0054] Optionally, the above-mentioned task exception handling method further includes:

[0055] By utilizing multiple historical log blocks included in the historical operation log, an exception tree with multiple exception levels is constructed.

[0056] Optionally, the construction includes an anomaly tree with multiple anomaly levels, including:

[0057] Based on the anomaly similarity threshold set at the initial anomaly level, multiple historical log blocks are clustered to form multiple first clusters containing at least one historical log block.

[0058] For each of the first clusters, an anomaly category is generated for the initial anomaly level based on the core information included in each historical log block of the first cluster.

[0059] For each exception level other than the initial exception level, perform the following operation:

[0060] When the previous anomaly level corresponding to the other anomaly level includes at least two anomaly categories, clustering is performed on the previous anomaly level that includes at least two anomaly categories;

[0061] If a second cluster is formed that satisfies the abnormal similarity threshold set for the other abnormal levels, an abnormal category is generated for the other abnormal levels based on at least two abnormal categories included in the second cluster.

[0062] Optionally, the above task exception handling method further includes: configuring exception keywords for each of the exception categories;

[0063] The step of providing the first anomaly category and the anomaly reason configured for the first anomaly category to the task manager includes:

[0064] Provide the task manager with the exception keywords configured for the first exception category and the exception reasons configured for the first exception category.

[0065] Optionally, the above-mentioned task exception handling method further includes: configuring a repair scheme for each of the exception categories;

[0066] The repair solution configured for the first anomaly category is provided to the task manager, or the repair solution is directly used to repair the task.

[0067] In a second aspect, embodiments of the present invention provide a task exception handling device, comprising:

[0068] The module includes a log processing module, an exception search module, and an interaction module.

[0069] The log processing module is used to obtain the running log corresponding to the task and split the running log into multiple log blocks;

[0070] The anomaly search module is used to match a first anomaly category for the log block from multiple anomaly levels included in a pre-constructed anomaly tree and at least one anomaly category included in each anomaly level; and to determine the anomaly reason configured for the first anomaly category corresponding to the log block according to the anomaly reason configured for each anomaly category in advance.

[0071] The interaction module is used to provide the first exception category and the exception reason configured for the first exception category to the task manager, so that the task manager can repair the task according to the first exception category and the exception reason configured for the first exception category.

[0072] One embodiment of the above invention has the following advantages or beneficial effects: by matching the first exception category to multiple log blocks split from the running log from multiple exception levels included in the pre-constructed exception tree and at least one exception category included in each exception level, and by determining the exception reason configured for the first exception category for multiple log blocks according to the exception reason configured for each exception category in advance, the exception category and exception reason of the task exception can be realized through intelligent log analysis, so as to detect the task exception in time and ensure the timeliness of task exception detection. The whole process does not require manual analysis of logs and other information, effectively reducing the manual cost of abnormal tasks.

[0073] Furthermore, by providing the task manager with the first anomaly category and the anomaly reason configured for the first anomaly category, the task manager can repair the task based on the first anomaly category and the anomaly reason configured for the first anomaly category, so that the abnormal task can be repaired in a timely manner, ensuring the timeliness of abnormal task identification and repair.

[0074] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0075] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0076] Figure 1 This is a schematic diagram of the main flow of the task exception handling method according to an embodiment of the present invention;

[0077] Figure 2 This is a schematic diagram of the main process for calculating the similarity between core information and the anomaly categories included in multiple anomaly levels according to an embodiment of the present invention;

[0078] Figure 3 This is a schematic diagram of the main process for filtering out the first anomaly category matching the log block according to an embodiment of the present invention;

[0079] Figure 4 This is a schematic diagram of the main process for constructing an anomaly tree including multiple anomaly levels according to an embodiment of the present invention;

[0080] Figure 5A This is a schematic diagram of the main flow of a task exception handling method according to an embodiment of the present invention;

[0081] Figure 5B This is a schematic diagram of the structure of the anomaly tree according to an embodiment of the present invention;

[0082] Figure 6 This is a schematic diagram of the main flow of the technical solution provided by another embodiment of the present invention;

[0083] Figure 7 This is a schematic diagram of the main modules of the task exception handling device according to an embodiment of the present invention;

[0084] Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0085] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0086] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0087] Figure 1 This is a schematic diagram of the main flow of a task exception handling method according to an embodiment of the present invention. Figure 1 As shown, the task exception handling method may include the following steps:

[0088] Step S101: Obtain the running log corresponding to the task and split the running log into multiple log blocks;

[0089] Among them, the operation log refers to the log generated by data processing tasks (such as data processing tasks such as extraction, transformation, fabrication, and push) in the database or data warehouse.

[0090] Step S102: Match the first exception category to multiple log blocks from the multiple exception levels included in the pre-constructed exception tree and the at least one exception category included in each exception level;

[0091] The exception category can be the core information of a single log block, or it can be a template or pattern of multiple log blocks obtained by clustering multiple log blocks (the template or pattern can reflect the core of the log block's exception). For example, a log block is: 2022-08-24T09:08:25.173+08:00][INFO]plumber.reader.mysql-reader.invoke(logging.clj 272)[clojure-agent-send-off-pool-2]: The MySQL database exception information is as follows: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure. Then, there exists an exception category in the heterogeneous tree: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure.

[0092] The terms "first" and "second" in the first and second anomaly categories are used to distinguish anomaly categories obtained at different stages or steps, and are not limitations on the anomaly categories themselves.

[0093] Step S103: Based on the exception reasons pre-configured for each exception category, determine the exception reasons configured for the first exception category for multiple log blocks.

[0094] For example, the exception category com.mysql.jdbc.exceptions.jdbc4.Communi-icationsException:Communications link failure is configured with the reason for the database connection failure.

[0095] Step S104: Provide the first exception category and the exception reason configured for the first exception category to the task manager so that the task manager can repair the task according to the first exception category and the exception reason configured for the first exception category.

[0096] exist Figure 1In the illustrated embodiment, by matching the first exception category to multiple log blocks split from the runtime log from multiple exception levels included in the pre-constructed exception tree and at least one exception category included in each exception level, and by determining the exception reason configured for the first exception category for each log block according to the exception reason configured for each exception category in advance, the exception category and exception reason of the task exception can be determined through intelligent log analysis, so as to detect the task exception in time and ensure the timeliness of task exception detection. The whole process does not require manual analysis of logs and other information, effectively reducing the manual cost of abnormal tasks.

[0097] Furthermore, by providing the task manager with the first anomaly category and the anomaly reason configured for the first anomaly category, the task manager can repair the task based on the first anomaly category and the anomaly reason configured for the first anomaly category, so that the abnormal task can be repaired in a timely manner, ensuring the timeliness of abnormal task identification and repair.

[0098] One specific implementation of step S101 may include: determining a log block as the log text located between every two adjacent log recording times based on the multiple log recording times included in the operation log. By determining a log block as the log text located between every two adjacent log recording times, the accuracy and completeness of anomaly category identification are ensured.

[0099] Furthermore, the initial anomaly level (such as...) among the multiple anomaly levels included in the above anomaly tree Figure 5B Each anomaly category included in the initial anomaly level (shown) is obtained through clustering of historical runtime logs; for each other anomaly level (such as the initial anomaly level)... Figure 5B The anomaly levels shown are: Level 2, Level 3, ..., and the highest level. Each anomaly category included in other anomaly levels is obtained by clustering the multiple anomaly categories included in the previous anomaly level. For example... Figure 5B The anomaly tree shown illustrates that, for the second-level anomaly hierarchy, the preceding anomaly hierarchy is the initial anomaly hierarchy. Therefore, the anomaly categories at the second-level are obtained by clustering the anomaly categories at the initial anomaly hierarchy. Similarly, for the third-level anomaly hierarchy, the preceding anomaly hierarchy is the second-level anomaly hierarchy. Thus, the anomaly categories at the third-level are obtained by clustering the anomaly categories at the second-level anomaly hierarchy. This anomaly tree ensures that most anomaly categories are clustered within it, effectively improving the intelligent identification of anomaly categories for tasks and guaranteeing the accuracy of the identified anomaly categories.

[0100] Furthermore, each anomaly level has a corresponding anomaly similarity threshold; for example... Figure 5BIn the various anomaly levels shown, the anomaly similarity threshold is set to 75% for the initial anomaly level, 65% for the second-level anomaly level, 45% for the third-level anomaly level, and so on, with the highest-level anomaly level having a similarity threshold of 5%. That is, the anomaly similarity thresholds set for each anomaly level in the anomaly tree generally start from the lowest anomaly level (…). Figure 5B From the initial anomaly level to higher anomaly levels (such as...) Figure 5B The highest anomaly level) decreases sequentially, i.e. Figure 5B The initial anomaly similarity threshold for the first anomaly level is greater than the second-level anomaly similarity threshold, the second-level anomaly similarity threshold is greater than the third-level anomaly similarity threshold, and so on. This setting effectively retrieves the corresponding anomaly category and cause for abnormal tasks. Accordingly, a specific implementation for matching the first anomaly category to multiple log blocks may include performing the following operations for each log block:

[0101] The process involves extracting core information from log blocks for clustering; calculating the similarity between this core information and the anomaly categories included in multiple anomaly levels; and filtering out the first anomaly category matching the log block based on the similarity and anomaly similarity thresholds set for the anomaly levels. This core information can be determined based on user-defined core features. Since the anomaly category is the core information of a specific log block, or it can be a template or pattern of multiple log blocks obtained by clustering multiple log blocks (this template or pattern can reflect the core anomalies of the log blocks), the similarity calculation can accurately filter out matching anomaly categories.

[0102] Specifically, such as Figure 2 As shown, the specific implementation of calculating the similarity between the core information and the anomaly categories included in multiple anomaly levels may include the following steps:

[0103] Step S201: Perform word segmentation on the core information to obtain one or more word groups;

[0104] This word segmentation process is implemented using existing word segmentation tools. For example, the core information MySQL database exception information is as follows: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:Communicationslink failure. The corresponding word segmentation phrases can be: MySQL, database, exception information, com, mysql, jdbc, exceptions, jdbc4, CommunicationsException, Communications link failure.

[0105] Step S202: When any exception level is the current exception level, calculate the maximum length of the string and the length of the longest substring of the core information based on one or more phrases and the exception categories included in the current exception level.

[0106] for example, Figure 5B Exception category 11 in the initial exception hierarchy shown: com.mysql.jdbc.exceptions.jdbc2.ExtractException: Extract failure; Figure 5B The exception category 12 in the initial exception hierarchy shown is: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications linkfailure.

[0107] The maximum string length can be calculated by taking multiple word groups obtained from word segmentation as one string and the exception category as another string, determining the number of words in both strings, and setting the maximum number of words as the maximum string length. For example, in the example above, the core information has 12 words (in the core information, one English word is considered one word, one Chinese word group is considered one word, and punctuation marks are not counted), and the exception category 11 has 8 words. Therefore, the maximum string length for the core information and exception category 11 is 12. Similarly, the exception category 12 has 9 words, so the maximum string length for the core information and exception category 12 is also 12.

[0108] The implementation of calculating the longest substring length of the core information is as follows: Determine the number of consecutively overlapping words between the segmented core information and the exception category. For example, for the consecutively overlapping words between the core information and exception category 11 in the example above: com mysql jdbc exceptions, the longest substring length of the core information is 4; and for the consecutively overlapping words between the core information and exception category 12 in the example above: com mysql jdbcexceptions jdbc4 CommunicationsException Communications link failure, the longest substring length of the core information is 9.

[0109] Step S203: Calculate the similarity between the core information and the anomaly categories included in the current anomaly level by using the maximum length of the string and the length of the longest substring of the core information.

[0110] The similarity can be calculated using the following formula (1).

[0111]

[0112] Where SSIM represents the similarity between the core information and the anomaly categories included in the previous anomaly level; L_sum represents the maximum string length; and L_max represents the similarity between anomaly categories.

[0113] The similarity between the core information given in the example above and anomaly category 11 is 33%, and the similarity between the core information and anomaly category 12 is 75%. Since the initial anomaly level similarity threshold is 75%, anomaly category 12 is an anomaly category that meets the similarity threshold for the core information mentioned above.

[0114] Furthermore, the specific implementation of filtering out the first exception category matching the log block mentioned above may include: performing operations such as... for multiple exception levels. Figure 3 Steps S301 to S306 shown:

[0115] Step S301: Locate the current exception level for core information from multiple exception levels;

[0116] In the initial screening phase, locating the current anomaly level is essentially locating the initial anomaly level.

[0117] Step S302: In the current anomaly level, search for the second anomaly category, wherein the similarity between the second anomaly category and the core information is greater than or equal to the anomaly similarity threshold corresponding to the current anomaly level;

[0118] For example, in the above example, the second anomaly category found in this step is: Figure 5B The anomaly category shown is 12.

[0119] Step S303: Generate a temporary exception category based on the core information of the log block and the second exception category;

[0120] The generation of temporary exception categories involves clustering the core information and the second exception category of the log block using existing clustering methods to obtain continuously overlapping strings. For example, in the example above, the continuously overlapping string between the core information and exception category 12 is "com mysql jdbc exceptions jdbc4 CommunicationsExceptionCommunications link failure". Then, based on this continuously overlapping string, the core information, and the exception category, backtracking is performed (i.e., replacing characters in different positions with *), resulting in two strings S_A and S_B. For example, backtracking using the continuously overlapping string and the aforementioned core information yields "***com mysql jdbc exceptions jdbc4CommunicationsException Communications link failure"; backtracking using the continuously overlapping string and the aforementioned exception category 12 still yields "commysql jdbc exceptions jdbc4CommunicationsException Communications link failure".Based on the string corresponding to the maximum length determined above (e.g., MySQL, database, exception information, com, mysql, jdbc, exceptions, jdbc4, CommunicationsException, Communications link failure in the example above), the shorter of the two strings S_A and S_B is padded to increase its length. For example, if the backtracked string obtained from exception category 12 is shorter, the backtracked string obtained from exception category 12 is padded to get ***com mysqljdbc exceptions jdbc4CommunicationsException Communications link failure. Then, the characters at the same positions in the longer backtracked string (e.g., ***com mysql jdbc exceptions jdbc4CommunicationsException Communications link failure obtained above) and the padded shorter string (e.g., ***com mysql jdbc exceptions jdbc4CommunicationsExceptionCommunications link failure obtained above) are compared. If all strings are consistent, the temporary exception category of the string commysql jdbc exceptions jdbc4 CommunicationsExceptionCommunications link failure is determined. In cases where strings are inconsistent, the inconsistent positions are replaced with * to obtain a temporary exception category. It is worth noting that the above comparison process can be performed from back to front. The new string obtained can be a reversed string. By reversing the reversed string, a temporary exception category is obtained. This temporary exception category is obtained by removing the leading * from the string. That is, the first character of the exception category or temporary exception category is a letter, while the * in the middle of the characters is retained.

[0121] Step S304: Compare the temporary anomaly category with the second anomaly category; if the comparison result indicates that the temporary anomaly category and the second anomaly category are consistent, then proceed to step S305; if the comparison result indicates that the temporary anomaly category and the second anomaly category are inconsistent, then proceed to step S306.

[0122] Step S305: Determine that the second exception category matches the first exception category of the log block, delete the temporary exception category, end the process and end the loop;

[0123] Step S306: Take the next exception level corresponding to the current exception level as the current exception level, and execute step S301.

[0124] Specifically, the specific implementation of step S303 above may include:

[0125] In cases where the current exception level is the initial exception level, the core information of the log block and the second exception category searched from the initial exception level are aggregated to form a temporary exception category;

[0126] The process of forming temporary exception categories in this aggregation process has been described in detail above and will not be repeated here.

[0127] For cases where the current anomaly level is other than the initial anomaly level, a temporary anomaly category corresponding to the previous anomaly level is determined (this temporary anomaly category is obtained by aggregating the core information with anomaly categories in the previous anomaly level that meet the similarity threshold of the previous anomaly level). The temporary anomaly category corresponding to the previous anomaly level and the second anomaly category found in the current anomaly level are then aggregated to form a temporary anomaly category. It is worth noting that the next anomaly level corresponding to the core information of the log block is only searched if no anomaly category corresponding to the core information of the log block is found in the current anomaly level.

[0128] Furthermore, targeting Figure 3 In the illustrated embodiment, after step S302 and before step S303, the method further includes: if no second anomaly category is found in step S302, directly executing step S306. This ensures that the anomaly category can be located for the task in the anomaly tree.

[0129] Furthermore, targeting Figure 3 The illustrated embodiments and related Figure 3 In the embodiments shown, following step S302, the method further includes: if no second anomaly category is found, generating a new anomaly category for the core information and adding the new anomaly category to the current anomaly level. This dynamic expansion of anomaly categories is achieved by adding new anomaly categories.

[0130] There are two specific ways to generate new exception categories for core information.

[0131] Specifically, the first concrete implementation of generating new exception categories for core information is to directly determine the core information as the new exception category.

[0132] The second specific implementation method for generating new exception categories for core information is as follows: For cases where the current exception level is another exception level than the initial exception level, determine the previous exception level corresponding to the other exception level; aggregate the temporary exception category generated by the previous exception level and the exception categories originally included in the previous exception level to form new exception categories for other exception levels.

[0133] Furthermore, targeting Figure 3 The illustrated embodiment, after step S304, further includes: if the comparison result indicates that the temporary exception category is inconsistent with the second exception category, adding the temporary exception category to the current exception level. This process directly adds the temporary exception category as a new exception category to the current exception level, dynamically expanding the exception categories during the process of matching exception categories for the task.

[0134] In addition, regarding Figure 3 In the illustrated embodiment, before step S306, the method further includes: if the comparison result indicates that the temporary exception category is inconsistent with the second exception category, determining whether the current exception level is the highest level of the exception tree; if so, ending the current process; otherwise, executing S306. This is to avoid entering an infinite loop during the exception category matching process and to ensure that the program can stop even if an exception category is not found.

[0135] Furthermore, after splitting the runtime log into multiple log blocks as described above, and before matching the first anomaly category to each log block, the process may further include: formatting each log block according to a preset log format; correspondingly, the specific implementation of obtaining core information for clustering from the log blocks may include: extracting core information from the formatted log blocks according to preset core features of the log blocks. Formatting can remove unnecessary information from the runtime log, thereby standardizing the log and making it easier to accurately extract core information.

[0136] Furthermore, the task exception handling method provided in the above embodiments may further include: constructing an exception tree including multiple exception levels using multiple historical log blocks included in the historical running log.

[0137] Specifically, such as Figure 4 As shown, a specific implementation of constructing an anomaly tree with multiple anomaly levels may include the following steps:

[0138] Step S401: Based on the anomaly similarity threshold set in the initial anomaly level, cluster multiple historical log blocks to form multiple first clusters containing at least one historical log block;

[0139] This clustering process is implemented using existing text clustering methods.

[0140] Step S402: For each first cluster, generate an anomaly category for the initial anomaly level based on the core information included in each historical log block of the first cluster;

[0141] That is, extract the core information template from the core information of each historical log block included in the first cluster (the core information template is obtained by retaining the common parts of the core information included in each historical log block, replacing the different characters or words with *, and then deleting the * before the first word), and use the core information template as the anomaly category.

[0142] Step S403: For each exception level other than the initial exception level, perform the following operation:

[0143] Step S4031: If the previous anomaly level corresponding to other anomaly levels includes at least two anomaly categories, cluster the previous anomaly level that includes at least two anomaly categories;

[0144] Step S4032: If a second cluster is formed that satisfies the abnormal similarity threshold set for other abnormal levels, anomaly categories are generated for other abnormal levels based on the at least two abnormal categories included in the second cluster.

[0145] The clustering process for other anomaly levels and the process for determining anomaly categories are the same as above, and will not be repeated here.

[0146] Furthermore, the above-mentioned task anomaly handling method also includes: configuring anomaly keywords for each anomaly category; correspondingly, the specific implementation of providing the first anomaly category and the anomaly reason configured for the first anomaly category to the task manager may include: providing the anomaly keywords configured for the first anomaly category and the anomaly reason configured for the first anomaly category to the task manager. Configuring anomaly keywords makes it easier for the task manager to identify task anomalies.

[0147] Furthermore, the above-mentioned task anomaly handling method also includes: configuring a repair plan for each anomaly category; providing the repair plan configured for the first anomaly category to the task manager, or directly using the repair plan to repair the task. Providing a repair plan makes it easier for the task manager to repair or automate the repair of anomaly tasks.

[0148] Figure 5A A schematic diagram of the main flow of another task exception handling method provided by an embodiment of the present invention is shown. Figure 5A As shown, the exception handling method for this task mainly includes the following steps:

[0149] Step S501: Determine the log format, the core features of the log block, and the anomaly similarity threshold set for each anomaly level, and provide them to steps S502 and S504 respectively;

[0150] This step can include user-input or custom log formats, core characteristics of log blocks, number of exception levels, and exception similarity thresholds set for each exception level.

[0151] For example, the defined log format can be: [ <time>] <host> <method> <content>.

[0152] Step S502: Construct an anomaly tree including multiple anomaly levels, and provide the anomaly tree for step S505;

[0153] This step mainly utilizes multiple historical log blocks from the historical runtime logs to construct an exception tree that includes multiple exception levels.

[0154] Specifically, based on the anomaly similarity threshold set at the initial anomaly level, multiple historical log blocks are clustered to form multiple first clusters containing at least one historical log block. For example, the historical logs include Log block1, Log block2, Log block3, ..., Log blockX. Clustering of multiple historical log blocks is primarily achieved using existing text clustering techniques. The similarity between the historical log blocks included in each first cluster is not lower than the anomaly similarity threshold set at the initial anomaly level (e.g., an anomaly similarity threshold of 95%). The resulting first clusters are: Cluster 1, Cluster 2, Cluster 3, ..., Cluster N. For each first cluster, based on the core information included in each historical log block within the first cluster, anomaly categories are generated for the initial anomaly level. For example, for Cluster 1, Cluster 2, Cluster 3, ..., Cluster N, the following categories are obtained: Figure 5B The initial anomaly hierarchy shown includes anomaly category 11, anomaly category 12, anomaly category 13, ..., anomaly category 1N, and anomaly category 1N+1.

[0155] For each other exception level besides the initial exception level (each other exception level is as follows) Figure 5B For the second-level, third-level, ..., highest-level anomaly (as shown), perform the following operations:

[0156] If the previous anomaly level corresponds to other anomaly levels and includes at least two anomaly categories, then clustering is performed on the previous anomaly level that includes at least two anomaly categories; Figure 5B Taking the second-level anomaly hierarchy as an example, the anomaly categories 11, 12, 13, ..., 1N, and 1N+1 of the anomaly hierarchy above the second-level anomaly hierarchy (i.e., the primary anomaly hierarchy) are clustered using existing text clustering techniques. Figure 5B Taking the three-level anomaly hierarchy as an example, existing text clustering techniques are used to cluster the anomaly categories 21, 22, 23, ..., 2M of the anomaly hierarchy above the three-level anomaly hierarchy (i.e., the two-level anomaly hierarchy).

[0157] If a second cluster is formed that satisfies the anomaly similarity threshold set for other anomaly levels, then anomaly categories are generated for other anomaly levels based on at least two anomaly categories included in the second cluster. Figure 5B Taking the second-level anomaly hierarchy as an example, after clustering the second-level anomaly hierarchy to generate a second cluster (the similarity between the anomaly categories included in the second cluster is not less than the anomaly similarity threshold set for the second-level anomaly hierarchy, for example, the anomaly similarity threshold is 85%), anomaly categories are generated to obtain... Figure 5B The second-level anomaly hierarchy shown includes anomaly categories 21, 22, 23, ..., 2M. Figure 5B Taking the three-level anomaly hierarchy as an example, after clustering the third-level anomaly hierarchy into a second cluster (the similarity between the anomaly categories included in the second cluster is not less than the anomaly similarity threshold set for the third-level anomaly hierarchy, for example, an anomaly similarity threshold of 65%), anomaly categories are generated to obtain... Figure 5B The three-level anomaly hierarchy shown includes anomaly category 31, anomaly category 32, anomaly category 33, ...

[0158] In summary, each anomaly category in the initial anomaly level of the anomaly tree is obtained through clustering of historical runtime logs; for the anomaly category other than the initial anomaly level (it is worth noting that the initial anomaly level involved in this embodiment of the invention is...), ... Figure 5B Each other anomaly level (other than the initial anomaly level shown) is (each of these other anomaly levels is as follows) Figure 5B The anomaly levels shown are: Level 2, Level 3, ..., Level 1 (highest anomaly level). Each anomaly category included in other anomaly levels is obtained by clustering the multiple anomaly categories included in the previous anomaly level corresponding to other anomaly levels.

[0159] In addition, exception keywords and repair solutions can be configured for each exception category.

[0160] Step S503: Obtain the execution log corresponding to the task;

[0161] The obtained runtime logs can be user-input runtime logs from the data warehouse or database, or runtime logs obtained directly from the log management terminal.

[0162] Step S504: Preprocess the runtime log according to the log format and the core characteristics of the log blocks;

[0163] This preprocessing can be achieved through the following steps S5401 to S5403.

[0164] Step S5041: Format the runtime log;

[0165] This step first determines a log block based on the multiple log record times included in the runtime log, identifying the log text between every two adjacent log record times; then, each log block is formatted according to a preset log format.

[0166] For example, a running log might look like this:

[0167] [2022-08-24T09:08:25.173+08:00][INFO]plumber.reader.mysql-reader.invoke(logging.clj 272)[clojure-agent-send-off-pool-2]: The following is the MySQL database exception information:

[0168] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

[0169] [2022-08-24T09:09:25.173+08:00]XXXXXXXXXXXXXXXXXXXXXXXXXX.

[0170] The log segment between the log recording time [2022-08-24T09:08:25.173+08:00] and the log recording time [2022-08-24T09:09:25.173+08:00] constitutes one log block.

[0171] [2022-08-24T09:08:25.173+08:00][INFO]plumber.reader.mysql-reader.invoke(logging.clj 272)[clojure-agent-send-off-pool-2]: The following is the MySQL database exception information:

[0172] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure.

[0173] For example, according to the log format defined above [ <time>] <host> <method> <content>Formatting the log blocks yields:

[0174] time:2022-08-24T09:08:25.173+08:00

[0175] host:INFO

[0176] method:plumber.reader.mysql-reader.invoke(logging.clj 272)[clojure-agent-send-off-pool-2]:

[0177] The following is a MySQL database error message:

[0178] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:

[0179] Communications link failure

[0180] The formatting process mainly involves each part ( <time>、 <level> 、 <method> 、 <content>The specific content of each part is obtained by using the terminator and the sequential relationship between the parts. For example, <time>This refers to the content within the first square brackets [] in the log block; for example, Content could be the text starting with "MySQL database" and ending with the last character of the log block. It's worth noting that this formatting process is just one example, and other methods can be used to distinguish the different parts of the format.

[0181] Step S5042: Extracting core information from formatted logs;

[0182] Based on the core characteristics of the pre-defined log blocks, extract core information from the formatted log blocks.

[0183] For example, the core characteristics of a predefined log block are as described above. <content>Then, the core information can be extracted from the formatted log block given in the example above: The MySQL database exception information is as follows:

[0184] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:

[0185] Communications link failure

[0186] Step S5043: Perform word segmentation on the core information;

[0187] This step of word segmentation can yield one or more word groups; this segmentation process can be implemented using existing word segmentation tools, which will not be elaborated here.

[0188] It is worth noting that the above steps S5401 to S5403 are performed for each log block.

[0189] Step S505: Perform multi-level clustering on the core information and the multiple anomaly levels included in the anomaly tree;

[0190] Specifically, for clustering processing, when any outlier level is the current outlier level, the following operations are performed:

[0191] Step S5051: Determine whether there is an exception category at the current exception level. If yes, proceed to step S5052; otherwise, proceed to step S5053.

[0192] by Figure 5B Taking the anomaly tree shown as an example, the initial anomaly level is first used as the current anomaly level. If no anomaly category is matched at the initial anomaly level, the second-level anomaly level is used as the current anomaly level. If no anomaly category is matched at the second-level anomaly level, the third-level anomaly level is used as the current anomaly level, and so on recursively.

[0193] Step S5052: Calculate the similarity between the core information and the anomaly category, and then proceed to step S5054;

[0194] The specific implementation of this similarity calculation:

[0195] Calculate the maximum length of the string and the length of the longest substring of the core information based on one or more phrases and the exception categories included in the current exception level;

[0196] The longest substring length refers to the length of characters or the number of consecutively overlapping words between the core information and the exception category.

[0197] Using the maximum string length and the longest substring length of the core information, calculate the similarity between the core information and the anomaly categories included in the previous anomaly levels.

[0198] The calculation process has been described above and will not be repeated here.

[0199] Step S5053: Generate a new exception category for the core information, and execute steps S5056 and S506 respectively;

[0200] Step S5054: Determine whether the similarity meets the abnormal similarity threshold. If yes, proceed to step S5055; otherwise, proceed to step S5053.

[0201] This step mainly involves comparing the similarity with an abnormal similarity threshold. If the similarity is less than the abnormal similarity threshold, it is determined that the similarity does not meet the abnormal similarity threshold; if the similarity is greater than or equal to the abnormal similarity threshold, it is determined that the similarity meets the abnormal similarity threshold.

[0202] Step S5055: Determine the anomaly category and the corresponding anomaly cause for the core information, and end the current process;

[0203] The specific implementation of step S5055 may include: generating a temporary anomaly category based on the core information of the log block and anomaly categories whose similarity meets the anomaly similarity threshold; specifically, for the case where the current anomaly level is the initial anomaly level, the core information of the log block and anomaly categories whose similarity meets the anomaly similarity threshold searched for at the initial anomaly level are aggregated to form a temporary anomaly category; for the case where the current anomaly level is another anomaly level besides the initial anomaly level, the temporary anomaly category corresponding to the previous anomaly level corresponding to the other anomaly level is determined, and the temporary anomaly category corresponding to the previous anomaly level and anomaly categories whose similarity meets the anomaly similarity threshold searched for at the current anomaly level are aggregated to form a temporary anomaly category. For example, the core information and Figure 5B If the similarity of anomaly category 13 shown meets the anomaly similarity threshold of the primary anomaly level, then the core information is aggregated with anomaly category 13 to obtain a temporary anomaly category; for example, the core information and... Figure 5B The similarity of the anomaly category 23 shown meets the anomaly similarity threshold of the second-level anomaly level. The temporary anomaly category corresponding to the previous anomaly level of the second-level anomaly level is determined (the temporary anomaly category is obtained by clustering the core information or the core information with the anomaly category in the previous anomaly level corresponding to the second-level anomaly level). The temporary anomaly category corresponding to the previous anomaly level and the anomaly category whose similarity meets the anomaly similarity threshold searched in the current anomaly level are aggregated to form a temporary anomaly category.

[0204] Compare temporary anomaly categories with anomaly categories whose similarity meets the anomaly similarity threshold;

[0205] If the comparison result indicates that the temporary anomaly category is consistent with the anomaly category whose similarity meets the anomaly similarity threshold, then the anomaly category whose similarity meets the anomaly similarity threshold is determined as the first anomaly category matching the log block, and the temporary anomaly category is deleted; if the comparison result indicates that the temporary anomaly category is inconsistent with the anomaly category whose similarity meets the anomaly similarity threshold, then a new anomaly category is generated for the core information, and the new anomaly category is added to the current anomaly level.

[0206] In addition, based on the exception reasons and exception keywords pre-configured for each exception category, the exception reasons and exception keywords configured for the first exception category can be determined for multiple log blocks; the exception keywords configured for the first exception category and the exception reasons configured for the first exception category are provided to the task manager.

[0207] Furthermore, a repair plan can be configured to provide the repair plan configured for the first anomaly category to the task manager, or the repair plan can be used directly to repair the task.

[0208] Step S5056: Determine whether the current exception level is the highest level of the exception tree. If so, end the current process; otherwise, proceed to step S5057.

[0209] Among them, the highest level is Figure 5B The highest level of anomaly shown.

[0210] Step S5057: Take the next exception level corresponding to the current exception level as the current exception level, and execute step S5051;

[0211] Step S506: Update the exception category;

[0212] Step S5061: Extract new exception categories for the current exception level of the exception tree;

[0213] The specific implementation of this step:

[0214] If the current exception level is the initial exception level, directly determine the core information as the new exception category;

[0215] For cases where the current exception level is any other than the initial exception level, determine the previous exception level corresponding to the other exception level; aggregate the temporary exception categories generated by the previous exception level and the exception categories originally included in the previous exception level to form new exception categories for the other exception levels.

[0216] Step S5062: Determine if the anomaly category in the anomaly tree has changed. If yes, proceed to step S5063; otherwise, end the current process.

[0217] Step S5063: Take the next exception level of the current exception level as the current exception level and execute step S5061.

[0218] It is worth noting that the above-mentioned exception category update is performed recursively for each exception level.

[0219] The above process enables intelligent identification of task anomalies and adds new anomaly categories to the anomaly tree to enrich the categories of task anomalies and further improve the accuracy of task anomaly identification.

[0220] In summary, as Figure 6 As shown, the framework of the technical solution provided by this embodiment of the invention mainly includes: Step S601: The user inputs the defined log format, defined log core information, number of levels, and anomaly similarity thresholds for each level, and obtains the input log; Step S602: Log preprocessing; Step S603: The core information of the log is clustered with multiple levels to obtain task anomaly categories; Step S604: New anomaly categories are extracted and updated for each level. The entire process only requires the user to define the log format, log core information, number of levels, and anomaly similarity thresholds for each level. The task anomaly identification process and the process of adding new task anomalies do not require user participation, realizing intelligent identification of task anomalies and updating of task anomaly categories, and effectively reducing labor costs.

[0221] Figure 7 This is a schematic diagram of the structure of a task exception handling device provided in an embodiment of the present invention. Figure 7 As shown, the task exception handling device 700 may include: a log processing module 701, an exception search module 702, and an interaction module 703, wherein...

[0222] The log processing module 701 is used to obtain the running log corresponding to the task and split the running log into multiple log blocks;

[0223] The anomaly search module 702 is used to match a first anomaly category for a log block from multiple anomaly levels included in a pre-built anomaly tree and at least one anomaly category included in each anomaly level; and to determine the anomaly reason configured for the first anomaly category corresponding to the log block according to the anomaly reason configured for each anomaly category in advance.

[0224] The interaction module 703 is used to provide the first exception category and the exception reason configured for the first exception category to the task manager, so that the task manager can repair the task according to the first exception category and the exception reason configured for the first exception category.

[0225] In this embodiment of the invention, the log processing module 701 is further configured to determine a log block as the log text located between every two adjacent log recording times based on the multiple log recording times included in the running log.

[0226] In this embodiment of the invention, each anomaly category included in the initial anomaly level among the multiple anomaly levels included in the anomaly tree is obtained by clustering historical operation logs; for each other anomaly level other than the initial anomaly level, each anomaly category included in the other anomaly level is obtained by clustering the multiple anomaly categories included in the previous anomaly level corresponding to the other anomaly level.

[0227] In this embodiment of the invention, each anomaly level is set with a corresponding anomaly similarity threshold; the anomaly search module 702 is further configured to, for each log block, perform the following steps: obtain core information from the log block for clustering; calculate the similarity between the core information and the anomaly categories included in multiple anomaly levels; and filter out the first anomaly category matching the log block based on the similarity and the anomaly similarity threshold set for the anomaly level.

[0228] In this embodiment of the invention, the anomaly search module 702 is further configured to perform word segmentation on the core information to obtain one or more word groups; and when any anomaly level is the current anomaly level, to perform calculation of the maximum length of the string and the length of the longest substring of the core information based on one or more word groups and the anomaly categories included in the current anomaly level; and to calculate the similarity between the core information and the anomaly categories included in the previous anomaly level using the maximum length of the string and the length of the longest substring of the core information.

[0229] In this embodiment of the invention, the anomaly search module 702 is further configured to cyclically perform the following operations N1 to N6 for multiple anomaly levels:

[0230] N1: Locate the current exception level for core information from multiple exception levels;

[0231] N2: In the current anomaly level, search for the second anomaly category, where the similarity between the second anomaly category and the core information is greater than or equal to the anomaly similarity threshold corresponding to the current anomaly level;

[0232] N3: Generate a temporary exception category based on the core information of the log block and the second exception category;

[0233] N4: Compare the temporary anomaly category with the second anomaly category; if the comparison result indicates that the temporary anomaly category and the second anomaly category are the same, then proceed to N5; if the comparison result indicates that the temporary anomaly category and the second anomaly category are not the same, then proceed to N6.

[0234] N5: Determine that the second exception category matches the first exception category of the log block, delete the temporary exception category, end the process and end the loop;

[0235] N6: Take the next exception level corresponding to the current exception level as the current exception level and execute operation N1.

[0236] In this embodiment of the invention, the anomaly search module 702 is further configured to, for the case where the current anomaly level is the initial anomaly level, aggregate the core information of the log block and the second anomaly category searched at the initial anomaly level to form a temporary anomaly category; and for the case where the current anomaly level is any other anomaly level besides the initial anomaly level, determine the temporary anomaly category corresponding to the previous anomaly level corresponding to the other anomaly level, and aggregate the temporary anomaly category corresponding to the previous anomaly level and the second anomaly category searched at the current anomaly level to form a temporary anomaly category.

[0237] In this embodiment of the invention, the anomaly search module 702 is further configured to directly execute N6 if N2 does not search for the second anomaly category.

[0238] In this embodiment of the invention, the anomaly search module 702 is further configured to generate a new anomaly category for the core information if no second anomaly category is found, and add the new anomaly category to the current anomaly level.

[0239] In this embodiment of the invention, the anomaly search module 702 is further configured to add the temporary anomaly category to the current anomaly level if the comparison result indicates that the temporary anomaly category is inconsistent with the second anomaly category.

[0240] In this embodiment of the invention, the anomaly search module 702 is further configured to determine whether the current anomaly level is the highest level of the anomaly tree when the comparison result indicates that the temporary anomaly category is inconsistent with the second anomaly category. If so, the current process ends; otherwise, N6 is executed.

[0241] In this embodiment of the invention, the anomaly search module 702 is further configured to directly determine the core information as a new anomaly category during the process of generating a new anomaly category for the core information.

[0242] In this embodiment of the invention, the anomaly search module 702 is further configured to, in the process of generating new anomaly categories for core information, determine the previous anomaly level corresponding to other anomaly levels when the current anomaly level is other than the initial anomaly level; and perform aggregation processing on the temporary anomaly categories generated by the previous anomaly level and the anomaly categories originally included in the previous anomaly level to form new anomaly categories for other anomaly levels.

[0243] In this embodiment of the invention, the log processing module 701 is further configured to format each log block according to a preset log format; and extract core information from the formatted log block according to the preset core features of the log block.

[0244] In embodiments of the present invention, such as Figure 7 As shown, the task exception handling device 700 may further include: an exception tree construction module 704, wherein,

[0245] The exception tree construction module 704 is used to construct an exception tree with multiple exception levels by utilizing multiple historical log blocks included in the historical runtime log.

[0246] In this embodiment of the invention, the anomaly tree construction module 704 is further configured to cluster multiple historical log blocks according to the anomaly similarity threshold set for the initial anomaly level, forming multiple first clusters containing at least one historical log block; for each first cluster, anomaly categories are generated for the initial anomaly level based on the core information included in each historical log block included in the first cluster; for each other anomaly level other than the initial anomaly level, the following operations are performed: if the previous anomaly level corresponding to the other anomaly level includes at least two anomaly categories, the previous anomaly level including at least two anomaly categories is clustered; if a second cluster that satisfies the anomaly similarity threshold set for the other anomaly level is clustered, anomaly categories are generated for the other anomaly levels based on the at least two anomaly categories included in the second cluster.

[0247] In this embodiment of the invention, each anomaly category is configured with anomaly keywords;

[0248] The interaction module 703 is further used to provide the task manager with the exception keywords configured in the first exception category and the exception reasons configured in the first exception category.

[0249] In this embodiment of the invention, a repair scheme is configured for each anomaly category;

[0250] The interaction module 703 is further used to provide the task manager with the repair solution configured for the first exception category.

[0251] In embodiments of the present invention, such as Figure 7 As shown, the task exception handling device 700 may further include: an exception repair module 705, wherein,

[0252] The exception repair module 705 is used to directly repair tasks using the repair scheme configured for the exception category.

[0253] Figure 8 An exemplary system architecture 800 is shown that can be applied to the task exception handling method or task exception handling apparatus of the present invention.

[0254] like Figure 8 As shown, the system architecture 800 may include terminal devices 801 and 802, a network 803, a log recording server 804, a data warehouse 805, a database 806, and a server 807. The network 803 serves as the medium for providing communication links between terminal devices 801 and 802 and server 807, between log recording server 804 and data warehouse 805, between log recording server 804 and database 806, between log recording server 804 and server 807, between data warehouse 805 and server 807, and between database 806 and server 807. The network 803 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0255] Users can use terminal devices 801 and 802 to interact with server 807 via network 803, and receive exception categories and reasons from server 807 through terminal devices 801 and 802. Various applications can be installed on terminal devices 801 and 802, such as web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0256] The logging server 804 can be a server that provides various services, such as a backend management server that supports unified management of the operation logs of the data warehouse 805 and the database 806 (for example only).

[0257] Data warehouse 805 and database 806 are servers that provide various services for data processing, such as data extraction, transformation, fabrication, and push, and store data before and after processing (for example only).

[0258] Server 807 can be a server that provides various services, such as a background management server that supports log analysis and task anomaly analysis (for example only). The background management server can process and analyze the logs recorded by the log server 804 and provide the processing and analysis results (such as the anomaly category and cause of the task - for example only) to the terminal devices 801 and 802.

[0259] Terminal devices 801 and 802 can be various electronic devices with displays and support for web browsing, including but not limited to desktop computers, smartphones, tablets, etc.

[0260] It should be noted that the task exception handling method provided in the embodiments of the present invention is generally executed by the server 807, and correspondingly, the task exception handling device is generally set in the server 807.

[0261] It should be understood that Figure 8 The number of terminal devices, networks, log servers, data warehouses, databases, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, log servers, data warehouses, databases, and servers can be included.

[0262] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing terminal devices or servers of the present invention. Figure 9 The terminal device or server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0263] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0264] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0265] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0266] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0267] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0268] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a log processing module, an exception search module, and an interaction module. The names of these modules do not necessarily limit the module itself; for example, the log processing module may also be described as "a module that splits runtime logs into multiple log blocks."

[0269] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to: acquire runtime logs corresponding to a task and split the runtime logs into multiple log blocks; match a first exception category to the multiple log blocks from multiple exception levels included in a pre-constructed exception tree and at least one exception category included in each exception level; determine the exception cause configured for the first exception category corresponding to the multiple log blocks according to the exception cause pre-configured for each exception category; and provide the first exception category and the exception cause configured for the first exception category to a task manager, so that the task manager can repair the task according to the first exception category and the exception cause configured for the first exception category.

[0270] According to the technical solution of the present invention, by matching a first exception category to multiple log blocks split from the running log from multiple exception levels included in a pre-constructed exception tree and at least one exception category included in each exception level, and by determining the exception reason configured for the first exception category for each log block according to the exception reason configured for each exception category in advance, the exception category and exception reason of the task exception can be determined through intelligent log analysis, so as to detect the task exception in a timely manner and ensure the timeliness of task exception detection. The whole process does not require manual analysis of logs and other information, effectively reducing the manual cost of abnormal tasks.

[0271] Furthermore, by providing the task manager with the first anomaly category and the anomaly reason configured for the first anomaly category, the task manager can repair the task based on the first anomaly category and the anomaly reason configured for the first anomaly category, so that the abnormal task can be repaired in a timely manner, ensuring the timeliness of abnormal task identification and repair.

[0272] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / content> < / time> < / content> < / method> < / level> < / time> < / content> < / method> < / host> < / time> < / content> < / method> < / host> < / time>

Claims

1. A method for handling task exceptions, characterized in that, include: Obtain the execution log corresponding to the task, and split the execution log into multiple log blocks; From the multiple anomaly levels included in the pre-constructed anomaly tree and the at least one anomaly category included in each anomaly level, a first anomaly category is matched for multiple log blocks; each anomaly category included in the starting anomaly level among the multiple anomaly levels included in the anomaly tree is obtained by clustering historical running logs; for each other anomaly level other than the starting anomaly level, each anomaly category included in the other anomaly level is obtained by clustering multiple anomaly categories included in the previous anomaly level corresponding to the other anomaly level; Based on the exception reasons pre-configured for each exception category, determine the exception reasons configured for the first exception category for each of the log blocks. The first exception category and the exception reason configured for the first exception category are provided to the task manager so that the task manager can repair the task based on the first exception category and the exception reason configured for the first exception category.

2. The task exception handling method according to claim 1, characterized in that, The step of splitting the operation log into multiple log blocks includes: determining a log block as the log text located between every two adjacent log record times, based on the multiple log record times included in the operation log.

3. The task exception handling method according to claim 1, characterized in that, Each of the aforementioned anomaly levels is configured with a corresponding anomaly similarity threshold; The first anomaly category is determined by matching multiple log blocks, including: For each of the log blocks, perform the following operations: Obtain the core information for clustering from the log block; Calculate the similarity between the core information and the anomaly categories included in the multiple anomaly levels; Based on the similarity and the anomaly similarity threshold set by the anomaly level, the first anomaly category matching the log block is selected.

4. The task exception handling method according to claim 3, characterized in that, The calculation of the similarity between the core information and the anomaly categories included in the multiple anomaly levels includes: The core information is segmented into words to obtain one or more word groups; and if any of the aforementioned exception levels is the current exception level, the following operations are performed: Calculate the maximum string length and the longest substring length of the core information based on one or more of the phrases and the exception categories included in the current exception level; Using the maximum length of the string and the length of the longest substring of the core information, calculate the similarity between the core information and the anomaly categories included in the previous anomaly level; And / or, After splitting the running log into multiple log blocks, and before matching the first exception category for the multiple log blocks, the method further includes: formatting each log block according to a preset log format. The process of obtaining core information for clustering from the log block includes: Based on the core characteristics of the pre-defined log blocks, extract core information from the formatted log blocks.

5. The task exception handling method according to claim 3, characterized in that, The filtering process for the first anomaly category matching the log block includes: For multiple exception levels, the following steps are performed repeatedly: From the multiple anomaly levels, locate the current anomaly level for the core information; In the current anomaly level, a second anomaly category is searched, wherein the similarity between the second anomaly category and the core information is greater than or equal to the anomaly similarity threshold corresponding to the current anomaly level; Based on the core information of the log block and the second exception category, a temporary exception category is generated; The temporary exception category is compared with the second exception category. If the comparison result indicates that the temporary exception category is consistent with the second exception category, the second exception category is determined to be the first exception category matching the log block, the temporary exception category is deleted, the process ends and the loop ends. If the comparison result indicates that the temporary exception category is inconsistent with the second exception category, the next exception level corresponding to the current exception level is taken as the current exception level, and the step of locating the current exception level for the core information from multiple exception levels is executed again.

6. The task exception handling method according to claim 5, characterized in that, The categories of generated temporary anomalies include: In the case where the current exception level is the initial exception level, the core information of the log block and the second exception category searched from the initial exception level are aggregated to form a temporary exception category; For cases where the current anomaly level is an anomaly level other than the initial anomaly level, a temporary anomaly category corresponding to the previous anomaly level is determined, and the temporary anomaly category corresponding to the previous anomaly level and the second anomaly category searched for at the current anomaly level are aggregated to form a temporary anomaly category. And / or, After searching for the second anomaly category and before generating the temporary anomaly category, the method further includes: if the second anomaly category is not found, directly taking the next anomaly level corresponding to the current anomaly level as the current anomaly level, and performing the step of locating the current anomaly level for the core information from multiple anomaly levels again; And / or, After comparing the temporary anomaly category with the second anomaly category, the method further includes: if the comparison result indicates that the temporary anomaly category is inconsistent with the second anomaly category, adding the temporary anomaly category to the current anomaly level; And / or, The task exception handling method also includes: If the comparison results indicate that the temporary anomaly category is inconsistent with the second anomaly category. Determine whether the current exception level is the highest level of the exception tree. If it is, end the current process; otherwise, execute the step of taking the next exception level corresponding to the current exception level as the current exception level.

7. The task exception handling method according to claim 5, characterized in that, Following the search for the second anomaly category, the following is also included: If the second anomaly category is not found, a new anomaly category is generated for the core information, and the new anomaly category is added to the current anomaly level.

8. The task exception handling method according to claim 7, characterized in that, The process of generating new anomaly categories for the core information includes: The core information was directly identified as a new anomaly category; or, For cases where the current exception level is any level other than the initial exception level, Determine the previous anomaly level corresponding to the other anomaly levels; The temporary exception categories generated at the previous exception level and the original exception categories included at the previous exception level are aggregated to form new exception categories for the other exception levels.

9. The task exception handling method according to claim 1, characterized in that, Also includes: Based on the anomaly similarity threshold set in the initial anomaly level, multiple historical log blocks included in the historical operation log are clustered to form multiple first clusters containing at least one historical log block. For each of the first clusters, an anomaly category is generated for the initial anomaly level based on the core information included in each historical log block of the first cluster. For each exception level other than the initial exception level, perform the following operation: When the previous anomaly level corresponding to the other anomaly level includes at least two anomaly categories, clustering is performed on the previous anomaly level that includes at least two anomaly categories; If a second cluster is formed that satisfies the abnormal similarity threshold set for the other abnormal levels, then based on the at least two abnormal categories included in the second cluster, abnormal categories are generated for the other abnormal levels, and an abnormal tree including multiple abnormal levels is constructed.

10. The task exception handling method according to claim 1, characterized in that, It also includes: each of the aforementioned anomaly categories is configured with anomaly keywords; The step of providing the first anomaly category and the anomaly reason configured for the first anomaly category to the task manager includes: providing the anomaly keyword configured for the first anomaly category and the anomaly reason configured for the first anomaly category to the task manager; And / or, The task anomaly handling method further includes: configuring a repair scheme for each anomaly category; providing the repair scheme configured for the first anomaly category to the task manager, or directly using the repair scheme to repair the task.

11. A task exception handling device, characterized in that, include: The module includes a log processing module, an exception search module, and an interaction module. The log processing module is used to obtain the running log corresponding to the task and split the running log into multiple log blocks; The anomaly search module is used to match a first anomaly category for the log block from multiple anomaly levels included in a pre-constructed anomaly tree and at least one anomaly category included in each anomaly level; each anomaly category included in the starting anomaly level among the multiple anomaly levels included in the anomaly tree is obtained by clustering historical running logs; for each other anomaly level other than the starting anomaly level, each anomaly category included in the other anomaly level is obtained by clustering multiple anomaly categories included in the previous anomaly level corresponding to the other anomaly level; and according to the anomaly reason pre-configured for each anomaly category, the anomaly reason configured for the first anomaly category corresponding to the log block is determined. The interaction module is used to provide the first exception category and the exception reason configured for the first exception category to the task manager, so that the task manager can repair the task according to the first exception category and the exception reason configured for the first exception category.

12. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images