Data processing method, device and equipment, readable storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a readable storage medium and a program product, which can be applied to the fields of cloud technology, artificial intelligence and the like, and the method comprises the following steps: extracting a plurality of key log data from log data; carrying out clustering processing on the plurality of key log data to obtain a plurality of target clustering sets, and determining task exception representation data corresponding to each target clustering set; based on the task exception representation data corresponding to each target clustering set, determining exception repair information corresponding to each target clustering set; and storing the task exception characterization data and the exception repair information corresponding to each target cluster set in a storage space in an associated manner, so that when the log data to be processed is matched with the target task exception characterization data in the storage space, the exception repair information associated with the target task exception characterization data in the storage space is output. The embodiment of the invention can improve the automation degree of operation and maintenance.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to data processing methods, data processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] With the continuous development of internet technology and the rapid increase in data volume, big data platforms have become key tools for processing and analyzing large-scale data. Currently, the operation and maintenance of big data platforms requires manually set detection rules to perform maintenance work on task nodes, such as detecting the running status of task nodes using preset detection rules. This method is inefficient, and with the continuous emergence of new components and services in big data platforms, a significant amount of manpower is needed to set detection rules for these new components and services, resulting in a low level of automation in operation and maintenance. Therefore, how to improve the automation level of operation and maintenance is an urgent problem to be solved. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, readable storage medium, and program product that can improve the automation level of operation and maintenance.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies.

[0006] Clustering is performed on multiple key log data to obtain multiple target cluster sets, and the task anomaly representation data corresponding to each target cluster set is determined.

[0007] Based on the task anomaly representation data corresponding to each target cluster set, determine the anomaly repair information corresponding to each target cluster set;

[0008] The task anomaly representation data and anomaly repair information corresponding to each target cluster set are associated and stored in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0009] On the other hand, embodiments of this application provide a data processing apparatus, which includes:

[0010] The extraction unit is used to extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies.

[0011] The processing unit is used to perform clustering processing on multiple key log data to obtain multiple target cluster sets, and to determine the task anomaly representation data corresponding to each target cluster set in the multiple target cluster sets;

[0012] The processing unit is also used to determine the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster;

[0013] The storage unit is used to associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0014] In one possible implementation, when the processing unit is used to perform clustering processing on multiple key log data to obtain multiple target cluster sets, it is specifically used for:

[0015] A first clustering process is performed on multiple key log data sets to obtain one or more initial cluster sets; each initial cluster set includes one or more key log data sets.

[0016] Perform a second clustering process on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set; any initial cluster set is any one of the one or more initial cluster sets.

[0017] In one possible implementation, when the processing unit performs the first clustering process on multiple key log data to obtain one or more initial cluster sets, it specifically performs the following:

[0018] Punctuation marks are extracted from multiple key log data to obtain the punctuation mark sequence corresponding to each key log data.

[0019] Based on the punctuation sequence corresponding to each key log data in multiple key log data sets, clustering is performed on the multiple key log data sets to obtain one or more initial cluster sets; the punctuation sequence corresponding to each key log data set included in each initial cluster set is the same.

[0020] In one possible implementation, when the processing unit performs a second clustering process on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set, it specifically performs the following:

[0021] Determine the text similarity between the first key log data and the second key log data in any initial cluster set; the first key log data is any key log data in any initial cluster set, and the second key log data is any key log data in any initial cluster set other than the first key log data.

[0022] Based on the text similarity between the first key log data and the second key log data, the key log data in any initial cluster set are clustered to obtain one or more target cluster sets corresponding to any initial cluster set.

[0023] In one possible implementation, when the processing unit is used to determine the task anomaly representation data corresponding to each of the multiple target cluster sets, it specifically performs the following:

[0024] Each key log data in any target cluster is segmented into words to obtain the first word segmentation result for each key log data in any target cluster; any target cluster can be any one of multiple target clusters.

[0025] The characters to be removed from the first word segmentation result of each key log data are removed to obtain the second word segmentation result of each key log data.

[0026] The word frequency statistics are obtained by performing word frequency statistics on the second word segmentation results of each key log data.

[0027] Based on the key log data in any target cluster set and the word segments to be replaced whose word frequency is less than the word frequency threshold in the word frequency statistics results, determine the task anomaly representation data corresponding to any target cluster set.

[0028] In one possible implementation, when the processing unit determines the task anomaly representation data corresponding to any target cluster set based on the key log data in any target cluster set and the word segments to be replaced whose word frequencies are less than the word frequency threshold in the word frequency statistics results, it specifically performs the following:

[0029] Replace the words to be replaced in any key log data in any target cluster with wildcards to obtain the task anomaly representation data corresponding to any target cluster.

[0030] Alternatively, replace the words to be replaced in each key log data in any target cluster with wildcards to obtain the replacement results for each key log data. Then, perform deduplication on the replacement results and determine one or more task anomaly representation data corresponding to any target cluster based on the deduplicated replacement results.

[0031] In one possible implementation, when the processing unit determines the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster, it specifically performs the following:

[0032] Get task prompt information; the task prompt information includes task content information and output format information;

[0033] The task anomaly representation data and task prompt information corresponding to each target cluster set are input into the large language model for processing to obtain the anomaly repair information corresponding to each target cluster set.

[0034] When the task content information includes output error message and output repair method message, the error repair information includes the error cause and repair method corresponding to the task error representation data.

[0035] In one possible implementation, the processing unit is also used for:

[0036] In response to a task diagnostic request from a terminal device, the target task is determined based on the task identifier carried in the task diagnostic request, and the corresponding log data to be processed is determined.

[0037] Retrieve anomaly representation data of multiple tasks stored in the storage space;

[0038] The log data to be processed is matched with the anomaly representation data of multiple tasks to obtain the matching results.

[0039] If the matching result indicates that the log data to be processed matches the target task anomaly representation data among multiple task anomaly representation data, then the anomaly repair information associated with the target task anomaly representation data is determined from the storage space.

[0040] Anomaly repair information associated with the target task anomaly representation data is returned to the terminal device.

[0041] Accordingly, embodiments of this application provide a computer device, which includes:

[0042] A processor is a tool for implementing computer programs.

[0043] A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method.

[0044] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0045] Accordingly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the data processing method described above.

[0046] This application extracts multiple key log data containing keywords indicating task anomalies from the log data recorded by task nodes during task execution, thereby efficiently locating anomalies. Then, clustering is performed on these key log data to obtain multiple target cluster sets, ensuring that the key log data in each target cluster set represents a specific anomaly type. Next, task anomaly representation data corresponding to each target cluster set is determined, and anomaly repair information is determined for each target cluster set based on this data, providing a targeted solution for each anomaly type. By associating and storing the task anomaly representation data and anomaly repair information corresponding to each target cluster set in a storage space, when the log data to be processed requiring anomaly detection matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output. This method improves the response speed and accuracy of anomaly repair processing, thereby increasing the efficiency and automation of operations and maintenance, while avoiding the need for manually setting detection rules, thus reducing operational costs. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application;

[0049] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0051] Figure 4A This is a schematic diagram illustrating a process for generating task anomaly representation data provided in an embodiment of this application;

[0052] Figure 4BThis is a schematic diagram of the overall process of clustering processing provided in an embodiment of this application;

[0053] Figure 4C This is a schematic diagram of a task diagnosis method provided in an embodiment of this application;

[0054] Figure 4D This is a schematic diagram of a knowledge maintenance interface provided in an embodiment of this application;

[0055] Figure 4E This is a schematic diagram of a task diagnosis interface provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] This application involves log data, pending log data, etc. When the above embodiments of this application are applied to specific products or technologies, the relevant data collection, use and processing processes should comply with the requirements of relevant laws and regulations. Before collecting data, the information processing rules should be informed and the individual consent of the subject should be obtained. Data should be processed in strict accordance with the requirements of laws and regulations and personal information processing rules, and technical measures should be taken to ensure the security of relevant data.

[0060] The data processing system provided in the embodiments of this application will now be described in conjunction with the accompanying drawings. The data processing system is suitable for implementing the data processing method provided in the embodiments of this application.

[0061] Please see Figure 1 This figure is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. The data processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network (LAN), a wide area network (WAN), or the mobile internet.

[0062] Terminal device 101 can also be referred to as a terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent, or user device. Terminal device can be a smart home appliance, a handheld device with wireless communication capabilities (e.g., a smartphone or tablet), a computing device (e.g., a personal computer (PC)), an in-vehicle terminal, a smart voice interaction device, a wearable device, or other smart devices, but is not limited to these. In this embodiment, terminal device 101 can refer to the equipment used by maintenance personnel to perform maintenance operations (such as task diagnosis of task nodes on a big data platform).

[0063] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In this embodiment, server 102 can refer to the management device corresponding to each task node, such as the backend server corresponding to the big data platform. Server 102 can provide task diagnostic services for each task node.

[0064] In one possible implementation, server 102 can extract multiple key log data from the log data recorded by the task nodes for task execution; then, it can perform clustering processing on the multiple key log data to obtain multiple target cluster sets, and determine the task anomaly representation data corresponding to each target cluster set (which can be used to represent the common characteristics of the key log data in each target cluster set); then, based on the task anomaly representation data corresponding to each target cluster set, it can determine the anomaly repair information corresponding to each target cluster set (which can be used to represent the anomaly cause and anomaly repair method of the anomaly type corresponding to the target cluster set); and finally, it can associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set in the storage space.

[0065] In one possible implementation, terminal device 101 can send a task diagnosis request carrying a task identifier (ID) to server 102 to detect whether the target task corresponding to the task identifier is abnormal. Server 102 can respond to the information query request, determine the target task based on the task identifier carried in the information query request, and determine the log data to be processed corresponding to the target task. Server 102 then obtains multiple task anomaly representation data stored in the storage space, performs matching processing on the log data to be processed and the multiple task anomaly representation data respectively, and obtains matching results. If the matching result indicates that the log data to be processed matches the target task anomaly representation data among the multiple task anomaly representation data, server 102 can determine the anomaly repair information associated with the target task anomaly representation data from the storage space, and return the anomaly repair information associated with the target task anomaly representation data to terminal device 101.

[0066] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, the data processing method provided in the embodiments of this application can be executed not only by server 102, but also by other servers or server clusters that are different from server 102 and can communicate with terminal device 101 and / or server 102. Those skilled in the art will understand that... Figure 1 The number of terminal devices and servers shown is merely illustrative. Any number of terminal devices and servers can be configured according to business needs. Furthermore, as system architecture evolves and new business scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems. In subsequent embodiments, "terminal device" will refer to the aforementioned terminal device 101, and "server" will refer to the aforementioned server 102; further details will not be repeated in subsequent embodiments.

[0067] The data processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0068] This application provides a data processing method that can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 2 As shown, the data processing method may include, but is not limited to, the following steps S201-S204:

[0069] S201. Extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies.

[0070] A task node is a node that executes a task. For example, it can be a task node in a big data platform used to execute big data tasks (such as Flink task nodes, Spark task nodes, MapReduce task nodes, and other different types of task nodes). These task nodes record various log information during task execution, such as task running status, performance indicators, and exception information, which helps in task diagnosis and performance optimization.

[0071] Log data typically includes multiple log lines, each representing a record of an event or operation. Critical log data refers to log lines that include keywords indicating task exceptions. These keywords can include terms such as "failed," "error," and "exception."

[0072] In this embodiment of the application, the computer device can acquire log data recorded by different types of task nodes (the number of task nodes of the same type can be one or more) for task execution, and then extract key log data from each acquired log data. For example, the log lines containing keywords used to indicate task abnormalities are extracted from each log data, thereby obtaining multiple key log data, so as to efficiently locate the abnormality and facilitate subsequent centralized data analysis and processing of multiple key log data.

[0073] S202. Perform clustering processing on multiple key log data to obtain multiple target cluster sets, and determine the task anomaly representation data corresponding to each target cluster set in the multiple target cluster sets.

[0074] In this embodiment of the application, the computer device can perform clustering processing on multiple key log data. Through clustering processing, multiple key log data can be grouped into cluster sets with similar characteristics (denoted as target cluster sets), which enables the key log data in each target cluster set to represent an anomaly type.

[0075] In this embodiment, the computer device can analyze the key log data included in each of multiple target cluster sets to obtain task anomaly representation data corresponding to each target cluster set. The task anomaly representation data indicates the common features extracted from the key log data included in each target cluster set, and is used to describe an anomaly type. For example, the computer device can extract common features from the key log data included in each target cluster set to obtain a regular expression corresponding to each target cluster set; this regular expression is the task anomaly representation data in this embodiment.

[0076] S203. Determine the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster.

[0077] In this embodiment of the application, the computer device can generate corresponding anomaly repair information for the task anomaly representation data corresponding to each target cluster set, thereby providing a targeted solution for each anomaly type.

[0078] In one possible implementation, the anomaly repair information may include one or both of the following: the cause of the anomaly corresponding to the task anomaly characterization data, and the anomaly repair method. By providing the cause of the anomaly and the anomaly repair method, it helps operations and maintenance personnel quickly identify the cause of the anomaly and provides a reference solution, thereby improving the efficiency of anomaly resolution.

[0079] S204. Associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0080] In this embodiment, the computer device can associate the task anomaly representation data and anomaly repair information corresponding to each target cluster set. That is, the task anomaly representation data and anomaly repair information have a one-to-one correspondence. Then, the associated task anomaly representation data and anomaly repair information are stored in a storage space (such as a database). Based on this, when the computer device subsequently obtains log data to be processed (referring to log data that currently needs anomaly detection), it can match the log data to be processed with each task anomaly representation data in the storage space. When the log data to be processed matches one or more task anomaly representation data (denoted as target task anomaly representation data) in the storage space, the anomaly repair information associated with the target task anomaly representation data can be obtained from the storage space, and the anomaly repair information associated with the target task anomaly representation data can be output.

[0081] In one possible implementation, after receiving a task diagnosis request from a terminal device, the computer device can check whether the log data to be processed matches the anomaly representation data of each task in the storage space. When the log data to be processed matches the anomaly representation data of the target task in the storage space, the computer device retrieves the anomaly repair information associated with the anomaly representation data of the target task in the storage space and returns the anomaly repair information associated with the anomaly representation data of the target task to the terminal device. This method allows the computer device to quickly retrieve and output the corresponding anomaly repair information after a successful anomaly match, while the terminal device can also quickly retrieve the anomaly repair information. This ensures efficient anomaly repair, reduces the impact of system anomalies, and thus improves the efficiency and automation of operation and maintenance.

[0082] In one possible implementation, the computer device can generate an anomaly identifier for each set of task anomaly characterization data and anomaly repair information; the anomaly identifier could be, for example, an error code. The computer device can then associate and store the anomaly identifier, task anomaly characterization data, and anomaly repair information in storage space (such as a database).

[0083] The above method extracts task anomaly representation data corresponding to different anomaly types from a large amount of log data, and stores the task anomaly representation data and corresponding anomaly repair information for different anomaly types in a knowledge base. If an anomaly type matching the task anomaly representation data stored in the knowledge base is detected in the log data to be processed, the anomaly repair information corresponding to that anomaly type can be quickly determined and output, thereby improving the response speed and accuracy of anomaly repair processing, thus improving the efficiency and automation of operation and maintenance, while avoiding the use of manual setting of detection rules for operation and maintenance work and reducing operation and maintenance costs.

[0084] Based on the above embodiments, the beneficial effects of this application are as follows: This application extracts multiple key log data containing keywords indicating task anomalies from the log data recorded by task nodes for task execution, thereby efficiently locating anomalies. Then, clustering is performed on these multiple key log data to obtain multiple target cluster sets, ensuring that the key log data in each target cluster set represents a specific anomaly type. Next, task anomaly representation data corresponding to each target cluster set is determined, and anomaly repair information corresponding to each target cluster set is determined based on this data, thus providing a targeted solution for each anomaly type. By associating and storing the task anomaly representation data and anomaly repair information corresponding to each target cluster set in a storage space, when the log data to be processed requiring anomaly detection matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output. This method can improve the response speed and accuracy of anomaly repair processing, thereby improving the efficiency and automation of operation and maintenance, while avoiding the need for manually setting detection rules, thus reducing operation and maintenance costs.

[0085] This application provides another data processing method, which can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 3 As shown, the data processing method may include, but is not limited to, the following steps S301-S308:

[0086] S301. Extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies.

[0087] In one possible implementation, the computer device can acquire log data recorded by different types of task nodes, such as Flink task nodes, Spark task nodes, and MapReduce task nodes. Then, it can extract key log data from the large amount of acquired log data, thereby covering anomaly types in different scenarios and ensuring the comprehensiveness and diversity of the extracted key log data. Furthermore, unified processing of log data from different types of task nodes helps to establish a cross-platform task diagnostic system, improving its versatility.

[0088] For example, the various big data components can run on the resource management system Yarn. Therefore, computer devices can request log data from each task node via the Hypertext Transfer Protocol (HTTP). For instance, an HTTP request could take the form of: http: / / ip:port / node / containerlogs / container_e317_1709652422096_62020797_01_000001 / tdwadmin / spark.log / ?start=-4096.

[0089] S302. Perform clustering processing on multiple key log data to obtain multiple target cluster sets.

[0090] In one possible implementation, for step S302 above, one implementation can be as follows: (1)-(2):

[0091] (1) Perform first clustering on multiple key log data to obtain one or more initial cluster sets; each initial cluster set includes one or more key log data.

[0092] In this embodiment of the application, the computer device can perform a first clustering process on multiple key log data. The first clustering process can be regarded as a preliminary clustering process, which can simplify the complexity of the data.

[0093] In one possible implementation, for step (1) above, one implementation can be as follows: steps ①-②:

[0094] ① Extract punctuation marks from multiple key log data sets to obtain the punctuation mark sequence corresponding to each key log data set.

[0095] In this embodiment of the application, the computer device can perform punctuation mark extraction processing on each key log data to obtain a punctuation mark sequence corresponding to each key log data. For example, punctuation marks (such as commas, periods, quotation marks, question marks, ellipses, etc.) can be extracted sequentially according to the characters in the key log data from front to back, and the sequentially extracted punctuation marks are used as a punctuation mark sequence.

[0096] ② Based on the punctuation sequence corresponding to each key log data in multiple key log data, perform clustering processing on multiple key log data to obtain one or more initial cluster sets; each key log data included in each initial cluster set has the same punctuation sequence.

[0097] In this embodiment of the application, the computer device can perform clustering processing on multiple key log data according to the punctuation sequence, and key log data with the same punctuation sequence are clustered into the same initial cluster set.

[0098] Since log data is typically generated based on predefined log templates, and these templates have certain characteristics in terms of format and structure—for example, the punctuation marks in log data generated from the same template are fixed and used to organize and separate log content—this embodiment of the application improves the efficiency and accuracy of clustering by performing clustering processing through punctuation mark sequences (i.e., identifying punctuation mark patterns in log data), thereby generating various initial cluster sets through preliminary clustering.

[0099] (2) Perform a second clustering process on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set; any initial cluster set is any one of the one or more initial cluster sets.

[0100] In this embodiment of the application, the computer device can perform a second clustering process on each initial cluster set obtained by the first clustering process. The second clustering process can be regarded as a refinement of the clustering process. The first clustering process simplifies the complexity of the data, while the second clustering process focuses on refining and optimizing the results of the first clustering process. Through the two clustering processes, the data can be clustered hierarchically, thereby improving the final clustering effect and ensuring the accuracy of the final target cluster sets.

[0101] In one possible implementation, for step (2) above, one implementation can be as follows: steps ①-②:

[0102] ① Determine the text similarity between the first key log data and the second key log data in any initial cluster set; the first key log data is any key log data in any initial cluster set, and the second key log data is any key log data in any initial cluster set other than the first key log data.

[0103] In this embodiment of the application, the computer device can calculate the text similarity between the first key log data and the second key log data in any initial cluster set. That is, the computer device needs to calculate the text similarity between each key log data in any initial cluster set and each of the other key log data, so as to perform clustering processing on the key log data in any initial cluster set according to the text similarity.

[0104] In one possible implementation, text similarity can be calculated using different algorithms, such as cosine similarity, Jaccard similarity, Euclidean distance, etc., or other algorithms can be used.

[0105] ② Based on the text similarity between the first key log data and the second key log data, perform clustering processing on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set.

[0106] In this embodiment of the application, the computer device can perform clustering processing on multiple key log data according to the text similarity between each key log data, thereby clustering similar key log data into the same initial cluster set.

[0107] In one possible implementation, the text similarity between two key log data sets can be calculated as follows: First, the two key log data sets are segmented into words, and characters to be removed (such as punctuation marks and numbers) in the segmentation results are removed, resulting in two data processing results; then, the vocabularies corresponding to the two data processing results are denoted as m and n, respectively, and the vocabulary that appears in both vocabulary m and vocabulary n is denoted as k; finally, the text similarity between the two key log data sets is calculated, denoted as S, and the calculation formula can be as follows:

[0108]

[0109] Where min{} represents finding the minimum value, and len(k) represents the number of elements in vocabulary k. This formula reflects the degree of overlap between two key log data at the vocabulary level. The closer its value is to 1, the more similar the two key log data are.

[0110] In one possible implementation, clustering based on text similarity can be performed as follows: Assuming any initial cluster set includes five key log data points, denoted as data 1, data 2, data 3, data 4, and data 5, the computer device can first calculate the text similarity between data 1 and data 2, data 3, data 4, and data 5, respectively. Data points in data 2, data 3, data 4, and data 5 with text similarity greater than or equal to a similarity threshold (e.g., 0.8) are grouped with data 1 in target cluster set 1. For example, data 1, data 2, and data 4 are grouped in target cluster set 1. Then, the ungrouped data is further processed. Specifically, the computer device calculates the text similarity between data 3 and data 5. If the text similarity is greater than or equal to the similarity threshold, data 3 and data 5 are grouped in target cluster set 2; if the text similarity is less than the similarity threshold, data 3 is grouped in target cluster set 2, and data 5 is grouped in target cluster set 3.

[0111] Based on the above steps (1)-(2), this application embodiment performs hierarchical clustering based on the two clustering dimensions of punctuation features and text similarity to obtain multiple target cluster sets, thus ensuring the accuracy of the clustering process.

[0112] It should be noted that the embodiments of this application can also perform clustering processing on multiple key log data in other ways to obtain multiple target cluster sets, such as using text clustering algorithms based on machine learning, etc., which will not be elaborated here.

[0113] S303. Perform word segmentation on each key log data in any target cluster set to obtain the first word segmentation result of each key log data in any target cluster set; any target cluster set can be any one of multiple target cluster sets.

[0114] In this embodiment of the application, taking any one of the multiple target cluster sets (denoted as any target cluster set) as an example, the computer device may perform word segmentation processing on each key log data in any target cluster set to obtain the first word segmentation result of each key log data.

[0115] For example, word segmentation can be achieved using tools such as Jieba, SnowNLP, THULAC, HanLP, and pkuseg, without limitation here.

[0116] S304. Remove the characters to be removed from the first word segmentation result of each key log data to obtain the second word segmentation result of each key log data.

[0117] In this embodiment, the computer device can perform a removal process on the characters to be removed in the first word segmentation result, thereby removing useless characters and helping to improve the accuracy of subsequent word frequency statistics processing and word segmentation replacement processing. For example, the characters to be removed may include one or more of punctuation marks and numbers.

[0118] S305. Perform word frequency statistics processing on the second word segmentation results of each key log data to obtain word frequency statistics results.

[0119] In this embodiment, the computer device can perform word frequency statistics processing on the second word segmentation result of each key log data to obtain word frequency statistics results, which include the frequency of each word appearing in the log data. Through word frequency statistics processing, word segments that appear frequently in the log data can be identified, thereby helping to identify the common features of this anomaly type.

[0120] S306. Based on the key log data in any target cluster set and the word segments to be replaced in the word frequency statistics results where the word frequency is less than the word frequency threshold, determine the task anomaly representation data corresponding to any target cluster set.

[0121] In this embodiment of the application, the computer device can determine the word segments whose word frequency is less than the word frequency threshold in the word frequency statistics results as the word segments that need to be replaced (denoted as the word segments to be replaced), and then perform the replacement process on the word segments to be replaced, thereby obtaining the task anomaly representation data corresponding to any target cluster set. This makes the task anomaly representation data represent the core features of the anomaly type without having to pay attention to the distinguishable features of the anomaly type in different scenarios.

[0122] It should be noted that the number of words to be replaced can be one or more. At the same time, the word frequency threshold can be flexibly set according to the business situation. For example, the word frequency threshold can be set to the maximum word frequency * 0.8, the maximum word frequency * 0.9, etc., and there is no limitation here.

[0123] In one possible implementation, for step S306 above, one implementation method is as follows: replace the word to be replaced in any key log data in any target cluster set with a wildcard to obtain the task anomaly representation data corresponding to any target cluster set.

[0124] For example, a computer device can arbitrarily select a key log data from any target cluster set, and then replace each word to be replaced in the key log data with a wildcard. The result of the replacement can be a regular expression, which is the task anomaly representation data in this embodiment. For example, the wildcard can be .*?, where .*? represents matching any number of arbitrary characters using a non-greedy matching method.

[0125] The above method only requires replacing any one key log data in each target cluster set to generate a unique task anomaly representation data for each target cluster set, without having to process all key log data in each target cluster set separately, thus improving the efficiency of task anomaly representation data generation.

[0126] In one possible implementation, for step S306 above, another implementation can be as follows: replace the word segment to be replaced in each key log data in any target cluster set with a wildcard to obtain the replacement result corresponding to each key log data, and perform deduplication processing on the replacement result, and determine one or more task anomaly representation data corresponding to any target cluster set based on the deduplicated replacement result.

[0127] For example, a computer device can perform replacement processing on each key log data in any target cluster set to obtain the replacement result corresponding to each key log data, and determine one or more task anomaly representation data through deduplication processing, which avoids generating duplicate task anomaly representation data.

[0128] The above method comprehensively considers all key log data in each target cluster set, and can generate one or more task anomaly representation data for each target cluster set, ensuring the comprehensiveness of the generated task anomaly representation data and avoiding the omission of anomaly types that would result in the failure to generate corresponding task anomaly representation data.

[0129] The following example illustrates the method for determining the task anomaly representation data corresponding to each target cluster in multiple target cluster sets: For example... Figure 4A As shown, Figure 4A This is a schematic diagram of a process for generating task anomaly representation data provided in an embodiment of this application. Taking any target cluster set including two key log data as an example, the two key log data are [Cause by com.xxx.TableNotFoundException:table a not found in b] and [Cause by com.xxx.TableNotFoundException:table c not found in d].

[0130] The computer device first performs word segmentation on the two key log data respectively to obtain the corresponding first word segmentation results, and then removes the characters to be removed (such as punctuation marks and numbers) in the two first word segmentation results to obtain two second word segmentation results. For example, the two second word segmentation results are [Cause by com xxxTableNotFoundException table a not found in b] and [Cause by com xxxTableNotFoundException table c not found in d].

[0131] The computer device then performs word frequency statistics on the two second word segmentation results to obtain the word frequency statistics result. For example, the word frequency statistics result can be [Cause:2,by:2,TableNotFoundException:2,…,c:1,d:1]. Here, Cause:2 indicates that the word frequency of Cause is 2. Simultaneously, the computer device can determine the maximum word frequency to calculate the word frequency threshold. For example, if the maximum word frequency is 2, then the word frequency threshold can be the maximum word frequency * 0.8, which is 1.6. Furthermore, the computer device can determine the word segments in the word frequency statistics result whose word frequencies are less than the word frequency threshold, which may include the following word segments: a, b, c, and d.

[0132] The computer device then determines the task anomaly representation data corresponding to any target cluster set based on two key log data sets and the word segments to be replaced in the word frequency statistics results whose word frequencies are less than the word frequency threshold. Specifically, the computer device performs replacement processing on the key log data, such as replacing the word segment to be replaced in any key log data with the wildcard .*? to obtain the task anomaly representation data. For example, the task anomaly representation data can be [Cause bycom.xxx.TableNotFoundException:table.*? not foundin.*?]. In this embodiment, the task anomaly representation data can be a regular expression.

[0133] The following example illustrates the overall process of clustering. Figure 4B As shown, Figure 4B This is a schematic diagram of the overall process of clustering processing provided in an embodiment of this application. The process can be as follows:

[0134] 401. Traverse each row in the log data set and extract key log data containing keywords used to indicate task anomalies.

[0135] 402. Perform word segmentation on each key log data.

[0136] 403. Extract the punctuation sequence of each key log data.

[0137] 404. Cluster the key log data according to the punctuation sequence to obtain the initial cluster set.

[0138] 405. Determine the text similarity between each key log data.

[0139] 406. Perform clustering processing on the initial cluster set according to text similarity to obtain the target cluster set.

[0140] 407. Determine the task anomaly representation data corresponding to each target cluster set.

[0141] The specific implementation methods of steps S401-S407 can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0142] S307. Determine the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster.

[0143] In one possible implementation, for step S307 above, one implementation can be as follows: (1)-(2):

[0144] (1) Obtain task prompt information; the task prompt information includes task content information and output format information.

[0145] In this embodiment, task prompt information is used to instruct the big oracle model on specific tasks that need to be processed. The task prompt information may include task content information and output format information. The task content information indicates the specific content of the task, such as indicating that the cause of the anomaly and the method for fixing it need to be output. The output format information indicates the output format requirements of the big oracle model, such as indicating that the results should be output according to a specific format specification.

[0146] For example, the task prompt information can be shown in Table 1 below:

[0147] Table 1

[0148]

[0149]

[0150] The section above ##Notice contains task content information, while the section below ##Notice contains output format information.

[0151] (2) Input the task anomaly representation data and task prompt information corresponding to each target cluster set into the large language model for processing to obtain the anomaly repair information corresponding to each target cluster set.

[0152] In this embodiment, the computer device can concatenate the task anomaly representation data corresponding to each target cluster set with the task prompt information to obtain input information. This input information is then fed into a large language model for processing, yielding the output of the large language model. Finally, anomaly repair information corresponding to each target cluster set is extracted from the output of the large language model. For example, the computer device can call OpenAI's interface to obtain the output of the large language model, which will strictly adhere to the Markdown format in the task prompt information.

[0153] Specifically, when the task content information includes output anomaly cause prompts and output repair method prompts, the anomaly repair information includes the anomaly cause and repair method corresponding to the task anomaly representation data. This method automatically generates the anomaly cause and repair method corresponding to different anomaly types in task anomaly representation data, reducing the workload of manual analysis and processing, improving the efficiency and accuracy of anomaly analysis and processing, and enhancing the automation level of operations and maintenance.

[0154] S308. Associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0155] In one possible implementation, after the computer device associates and stores the task anomaly representation data and anomaly repair information corresponding to each target cluster set into the storage space (that is, after associating and storing the task anomaly representation data and anomaly repair information generated based on a large amount of log data into the storage space), the computer device can perform task diagnosis based on the data stored in the storage space (that is, the knowledge base).

[0156] Based on this, the computer device can also perform the following steps (1)-(5):

[0157] (1) In response to the task diagnosis request of the terminal device, determine the target task based on the task identifier carried in the task diagnosis request, and determine the log data to be processed corresponding to the target task.

[0158] In this embodiment, the terminal device can send a task diagnostic request to the computer device. The task diagnostic request is used to indicate whether an anomaly occurs during the execution of a certain task node. Then, the computer device can determine the target task based on the task identifier (such as task ID) carried in the task diagnostic request, and determine the log data to be processed corresponding to the target task.

[0159] (2) Obtain multiple task anomaly representation data stored in the storage space.

[0160] (3) Match the log data to be processed with the anomaly representation data of multiple tasks respectively to obtain the matching results.

[0161] In steps (2)-(3) above, the computer device needs to retrieve all task anomaly representation data stored in the storage space for anomaly matching. Then, the log data to be processed and all task anomaly representation data stored in the storage space are matched separately to find matching task anomaly representation data (denoted as target task anomaly representation data). If target task anomaly representation data is found, it means that the target task has encountered this type of anomaly during task execution.

[0162] (4) If the matching result indicates that the log data to be processed matches the target task anomaly representation data among multiple task anomaly representation data, then determine the anomaly repair information associated with the target task anomaly representation data from the storage space.

[0163] (5) Return the anomaly repair information associated with the target task anomaly representation data to the terminal device.

[0164] The above method matches the log data to be processed with the task anomaly representation data stored in the knowledge base, thereby quickly locating the anomalies in the target task from the knowledge base, improving the efficiency of task diagnosis, avoiding manual intervention by operation and maintenance personnel, and increasing the degree of automation of operation and maintenance.

[0165] It should be noted that there can be multiple target task anomaly representation data. In other words, the computer device can match multiple types of anomalies existing in the target task. Then, the computer device can return multiple target task anomaly representation data and their respective associated anomaly repair information to the terminal device. The operation and maintenance personnel can perform corresponding repair processing based on the anomaly repair information returned by the computer device, which can greatly improve the anomaly repair efficiency.

[0166] The task diagnosis method provided in the embodiments of this application will be illustrated below with examples: Figure 4C As shown, Figure 4CThis is a schematic diagram of a task diagnosis method provided in an embodiment of this application: First, a computer device can collect log data from different types of task nodes, such as Flink task nodes, Spark task nodes, and MapReduce task nodes. Then, a mining program (which includes processing logic such as clustering and determining the task anomaly representation data corresponding to each target cluster set) mines task anomaly representation data corresponding to different anomaly types. The task anomaly representation data is then input into a large language model for processing to obtain anomaly repair information. The task anomaly representation data and anomaly repair information are then associated and stored in a knowledge base. Furthermore, the computer device can act as a task diagnosis server, providing data maintenance functions for the knowledge base and task diagnosis functions. The task diagnosis function can refer to outputting anomaly repair information associated with the target task anomaly representation data in the storage space when the log data to be processed matches the target task anomaly representation data in the storage space; this will not be elaborated further here.

[0167] Specifically, for the data maintenance function, the mined task anomaly representation data (such as error codes) can be stored in MySQL. This data can be accessed via a simple Java service, and the front-end interface provides options for disabling, enabling, and modifying the task anomaly representation data in the knowledge base. For the task diagnosis function, after the user inputs a task identifier (ID), log data can be retrieved by concatenating the task identifier with a uniform resource locator (URL). Then, each task anomaly representation data in the knowledge base is traversed; if the log data matches a task anomaly representation data in the knowledge base, the matched task anomaly representation data and corresponding anomaly repair information are returned.

[0168] like Figure 4D As shown, Figure 4DThis is a schematic diagram of a knowledge maintenance interface provided in an embodiment of this application. The data maintenance interface includes multiple pieces of knowledge, each of which refers to data related to a task's anomaly representation data. Specifically, each piece of knowledge includes the system name, error code (referring to the system tag information corresponding to the generated error keyword, which is more concise than the error keyword for easier retrieval), error keyword (i.e., task anomaly representation data), level (e.g., minor, severe), status (e.g., normal, disabled), cause of anomaly, detection rate, effectiveness rate, creator, etc. In addition, each piece of knowledge also provides various operation controls, such as view, modify, and delete operation controls. When the user clicks the view operation control, they can also see the anomaly repair method corresponding to that piece of knowledge. At the same time, the knowledge maintenance interface also provides an error code retrieval function to improve retrieval efficiency. In addition, the knowledge maintenance interface also includes a new error code control, which is used to add knowledge obtained through manual processing (e.g., knowledge obtained through expert analysis).

[0169] like Figure 4E As shown, Figure 4E This is a schematic diagram of a task diagnosis interface provided in an embodiment of this application. The task diagnosis interface may include one or both of the following: manual diagnosis results (such as diagnosis results corresponding to knowledge obtained from expert analysis) and AI diagnosis results (such as diagnosis results stored in storage space). For example, manual diagnosis results may include a level (such as severe), diagnosis rules (such as error codes), nodes (such as WeData-DE), causes, and suggestions, wherein the causes and suggestions are based on expert analysis. For example, AI diagnosis results may include system name, error keywords, causes of anomalies, and methods for repairing anomalies, which will not be elaborated here.

[0170] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0171] Please see Figure 5 This figure is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. This data processing device can be installed in the computer equipment provided in the embodiment of this application, and the computer equipment can be as described above. Figure 1 Server 102 in the data processing system shown. Figure 5 The data processing device shown can be a computer program running on a computer device, which can be used to execute... Figure 2 or Figure 3 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 5 The data processing apparatus may include the following units:

[0172] Extraction unit 501 is used to extract multiple key log data from the log data recorded by the task node for task execution, and each key log data includes keywords used to indicate task anomalies.

[0173] The processing unit 502 is used to perform clustering processing on multiple key log data to obtain multiple target cluster sets, and to determine the task anomaly representation data corresponding to each target cluster set in the multiple target cluster sets;

[0174] The processing unit 502 is also used to determine the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster;

[0175] Storage unit 503 is used to associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0176] In one possible implementation, when processing unit 502 performs clustering processing on multiple key log data to obtain multiple target cluster sets, it specifically performs the following:

[0177] A first clustering process is performed on multiple key log data sets to obtain one or more initial cluster sets; each initial cluster set includes one or more key log data sets.

[0178] Perform a second clustering process on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set; any initial cluster set is any one of the one or more initial cluster sets.

[0179] In one possible implementation, when processing unit 502 performs a first clustering process on multiple key log data to obtain one or more initial cluster sets, it specifically performs the following:

[0180] Punctuation marks are extracted from multiple key log data to obtain the punctuation mark sequence corresponding to each key log data.

[0181] Based on the punctuation sequence corresponding to each key log data in multiple key log data sets, clustering is performed on the multiple key log data sets to obtain one or more initial cluster sets; the punctuation sequence corresponding to each key log data set included in each initial cluster set is the same.

[0182] In one possible implementation, when processing unit 502 performs a second clustering process on key log data in any initial clustering set to obtain one or more target clustering sets corresponding to any initial clustering set, it specifically performs the following:

[0183] Determine the text similarity between the first key log data and the second key log data in any initial cluster set; the first key log data is any key log data in any initial cluster set, and the second key log data is any key log data in any initial cluster set other than the first key log data.

[0184] Based on the text similarity between the first key log data and the second key log data, the key log data in any initial cluster set are clustered to obtain one or more target cluster sets corresponding to any initial cluster set.

[0185] In one possible implementation, when determining the task anomaly representation data corresponding to each of the multiple target cluster sets, the processing unit 502 specifically performs the following:

[0186] Each key log data in any target cluster is segmented into words to obtain the first word segmentation result for each key log data in any target cluster; any target cluster can be any one of multiple target clusters.

[0187] The characters to be removed from the first word segmentation result of each key log data are removed to obtain the second word segmentation result of each key log data.

[0188] The word frequency statistics are obtained by performing word frequency statistics on the second word segmentation results of each key log data.

[0189] Based on the key log data in any target cluster set and the word segments to be replaced whose word frequency is less than the word frequency threshold in the word frequency statistics results, determine the task anomaly representation data corresponding to any target cluster set.

[0190] In one possible implementation, when processing unit 502 determines the task anomaly representation data corresponding to any target cluster based on key log data in any target cluster and word segments to be replaced whose word frequency is less than the word frequency threshold in the word frequency statistics results, it specifically performs the following:

[0191] Replace the words to be replaced in any key log data in any target cluster with wildcards to obtain the task anomaly representation data corresponding to any target cluster.

[0192] Alternatively, replace the words to be replaced in each key log data in any target cluster with wildcards to obtain the replacement results for each key log data. Then, perform deduplication on the replacement results and determine one or more task anomaly representation data corresponding to any target cluster based on the deduplicated replacement results.

[0193] In one possible implementation, when processing unit 502 determines the anomaly repair information corresponding to each target cluster set based on the task anomaly representation data corresponding to each target cluster set, it specifically performs the following:

[0194] Get task prompt information; the task prompt information includes task content information and output format information;

[0195] The task anomaly representation data and task prompt information corresponding to each target cluster set are input into the large language model for processing to obtain the anomaly repair information corresponding to each target cluster set.

[0196] When the task content information includes output error message and output repair method message, the error repair information includes the error cause and repair method corresponding to the task error representation data.

[0197] In one possible implementation, the processing unit 502 is further configured to:

[0198] In response to a task diagnostic request from a terminal device, the target task is determined based on the task identifier carried in the task diagnostic request, and the corresponding log data to be processed is determined.

[0199] Retrieve anomaly representation data of multiple tasks stored in the storage space;

[0200] The log data to be processed is matched with the anomaly representation data of multiple tasks to obtain the matching results.

[0201] If the matching result indicates that the log data to be processed matches the target task anomaly representation data among multiple task anomaly representation data, then the anomaly repair information associated with the target task anomaly representation data is determined from the storage space.

[0202] Anomaly repair information associated with the target task anomaly representation data is returned to the terminal device.

[0203] It should be noted that the functions of each unit of the data processing device in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the method embodiments of this application, which will not be repeated here.

[0204] According to another embodiment of this application, Figure 5The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0205] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 3 Computer programs for the steps involved in some or all of the methods shown, to construct, for example... Figure 5 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0206] Based on the above methods and apparatus embodiments, this application provides a computer device. Please refer to... Figure 6 This figure is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6 The computer device shown includes at least a processor 601, an input interface 602, an output interface 603, and a computer-readable storage medium 604. The processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 can be connected via a bus or other means.

[0207] Computer-readable storage medium 604 can be stored in the memory of a computer device. Computer-readable storage medium 604 is used to store computer programs, including program instructions. Processor 601 is used to execute the computer program stored in computer-readable storage medium 604. Processor 601 (or CPU (Central Processing Unit)) is the computing and control core of the computer device; it is suitable for implementing computer programs, specifically for loading and executing computer programs to achieve the above-mentioned functions. Figure 2 or Figure 3 The method flow is shown.

[0208] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space for storing the operating system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by a processor. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0209] The computer equipment can be the above. Figure 1 The data processing system shown includes server 102. In a specific implementation, processor 601 loads and executes a computer program stored in computer-readable storage medium 604 to implement the corresponding steps of the data processing methods in the various method embodiments of this application. Specifically, the computer program in computer-readable storage medium 604 is loaded by processor 601 and executed with the following steps:

[0210] Extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies.

[0211] Clustering is performed on multiple key log data to obtain multiple target cluster sets, and the task anomaly representation data corresponding to each target cluster set is determined.

[0212] Based on the task anomaly representation data corresponding to each target cluster set, determine the anomaly repair information corresponding to each target cluster set;

[0213] The task anomaly representation data and anomaly repair information corresponding to each target cluster set are associated and stored in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

[0214] In one possible implementation, when processor 601 performs clustering processing on multiple key log data to obtain multiple target cluster sets, it specifically performs the following:

[0215] A first clustering process is performed on multiple key log data sets to obtain one or more initial cluster sets; each initial cluster set includes one or more key log data sets.

[0216] Perform a second clustering process on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set; any initial cluster set is any one of the one or more initial cluster sets.

[0217] In one possible implementation, when processor 601 performs a first clustering process on multiple key log data to obtain one or more initial cluster sets, it specifically performs the following:

[0218] Punctuation marks are extracted from multiple key log data to obtain the punctuation mark sequence corresponding to each key log data.

[0219] Based on the punctuation sequence corresponding to each key log data in multiple key log data sets, clustering is performed on the multiple key log data sets to obtain one or more initial cluster sets; the punctuation sequence corresponding to each key log data set included in each initial cluster set is the same.

[0220] In one possible implementation, when processor 601 performs a second clustering process on key log data in any initial cluster set to obtain one or more target cluster sets corresponding to any initial cluster set, it specifically performs the following:

[0221] Determine the text similarity between the first key log data and the second key log data in any initial cluster set; the first key log data is any key log data in any initial cluster set, and the second key log data is any key log data in any initial cluster set other than the first key log data.

[0222] Based on the text similarity between the first key log data and the second key log data, the key log data in any initial cluster set are clustered to obtain one or more target cluster sets corresponding to any initial cluster set.

[0223] In one possible implementation, when determining the task anomaly representation data corresponding to each of the multiple target cluster sets, the processor 601 specifically performs the following:

[0224] Each key log data in any target cluster is segmented into words to obtain the first word segmentation result for each key log data in any target cluster; any target cluster can be any one of multiple target clusters.

[0225] The characters to be removed from the first word segmentation result of each key log data are removed to obtain the second word segmentation result of each key log data.

[0226] The word frequency statistics are obtained by performing word frequency statistics on the second word segmentation results of each key log data.

[0227] Based on the key log data in any target cluster set and the word segments to be replaced whose word frequency is less than the word frequency threshold in the word frequency statistics results, determine the task anomaly representation data corresponding to any target cluster set.

[0228] In one possible implementation, when processor 601 determines the task anomaly representation data corresponding to any target cluster based on key log data in any target cluster and word segments to be replaced whose word frequencies are less than the word frequency threshold in the word frequency statistics results, it specifically performs the following:

[0229] Replace the words to be replaced in any key log data in any target cluster with wildcards to obtain the task anomaly representation data corresponding to any target cluster.

[0230] Alternatively, replace the words to be replaced in each key log data in any target cluster with wildcards to obtain the replacement results for each key log data. Then, perform deduplication on the replacement results and determine one or more task anomaly representation data corresponding to any target cluster based on the deduplicated replacement results.

[0231] In one possible implementation, when the processor 601 determines the anomaly repair information corresponding to each target cluster set based on the task anomaly representation data corresponding to each target cluster set, it specifically performs the following:

[0232] Get task prompt information; the task prompt information includes task content information and output format information;

[0233] The task anomaly representation data and task prompt information corresponding to each target cluster set are input into the large language model for processing to obtain the anomaly repair information corresponding to each target cluster set.

[0234] When the task content information includes output error message and output repair method message, the error repair information includes the error cause and repair method corresponding to the task error representation data.

[0235] In one possible implementation, processor 601 is also used for:

[0236] In response to a task diagnostic request from a terminal device, the target task is determined based on the task identifier carried in the task diagnostic request, and the corresponding log data to be processed is determined.

[0237] Retrieve anomaly representation data of multiple tasks stored in the storage space;

[0238] The log data to be processed is matched with the anomaly representation data of multiple tasks to obtain the matching results.

[0239] If the matching result indicates that the log data to be processed matches the target task anomaly representation data among multiple task anomaly representation data, then the anomaly repair information associated with the target task anomaly representation data is determined from the storage space.

[0240] Anomaly repair information associated with the target task anomaly representation data is returned to the terminal device.

[0241] In specific implementations, the processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 3 The implementation methods described in the relevant embodiments of the provided method can also be used to execute the embodiments of this application. Figure 6 The implementation methods described in the relevant embodiments of the provided device will not be repeated here.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, systems, and computer devices can be implemented in other ways. The embodiments described above are merely illustrative, and the division of units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0243] This application also provides a computer program product, which includes program instructions stored in a computer-readable storage medium. A processor of a computer device reads the program instructions from the computer-readable storage medium and executes the program instructions, causing the computer device to perform the aforementioned data processing method, which will not be described in detail here.

[0244] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in or transmitted through a computer-readable storage medium. The program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0246] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0247] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0248] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies. Clustering is performed on the multiple key log data to obtain multiple target cluster sets, and the task anomaly representation data corresponding to each target cluster set is determined. Based on the task anomaly representation data corresponding to each target cluster set, determine the anomaly repair information corresponding to each target cluster set; The task anomaly representation data and anomaly repair information corresponding to each target cluster set are associated and stored in the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

2. The method as described in claim 1, characterized in that, The clustering process performed on the multiple key log data results in multiple target cluster sets, including: A first clustering process is performed on the multiple key log data to obtain one or more initial cluster sets; each initial cluster set includes one or more key log data sets. A second clustering process is performed on the key log data in any initial cluster set to obtain one or more target cluster sets corresponding to the initial cluster set; the initial cluster set is any one of the one or more initial cluster sets.

3. The method as described in claim 2, characterized in that, The first clustering process is performed on the multiple key log data to obtain one or more initial cluster sets, including: Punctuation mark extraction is performed on the multiple key log data to obtain the punctuation mark sequence corresponding to each key log data in the multiple key log data; Based on the punctuation sequence corresponding to each key log data in the plurality of key log data, the plurality of key log data are clustered to obtain one or more initial cluster sets; each key log data included in each initial cluster set has the same punctuation sequence.

4. The method as described in claim 2, characterized in that, The second clustering process, which involves performing a second clustering operation on key log data in any initial clustering set to obtain one or more target clustering sets corresponding to any initial clustering set, includes: Determine the text similarity between a first key log data and a second key log data in any initial cluster set; the first key log data is any key log data in the initial cluster set, and the second key log data is any key log data in the initial cluster set other than the first key log data. Based on the text similarity between the first key log data and the second key log data, the key log data in any initial cluster set are clustered to obtain one or more target cluster sets corresponding to any initial cluster set.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the task anomaly representation data corresponding to each of the plurality of target cluster sets includes: Each key log data in any target cluster set is segmented into words to obtain the first word segmentation result for each key log data in any target cluster set; the any target cluster set is any one of the plurality of target cluster sets; The characters to be removed from the first word segmentation result of each key log data are removed to obtain the second word segmentation result of each key log data. The second word segmentation result of each key log data is subjected to word frequency statistics processing to obtain the word frequency statistics result; Based on the key log data in any target cluster set and the word segments to be replaced in the word frequency statistics results where the word frequency is less than the word frequency threshold, the task anomaly representation data corresponding to any target cluster set is determined.

6. The method as described in claim 5, characterized in that, The step of determining the task anomaly representation data corresponding to any target cluster based on key log data in any target cluster and word segments to be replaced with word frequencies less than the word frequency threshold in the word frequency statistics results includes: Replace the word segment to be replaced in any key log data in any target cluster set with a wildcard to obtain the task anomaly representation data corresponding to any target cluster set; Alternatively, replace the words to be replaced in each key log data in any of the target clusters with wildcards to obtain the replacement result corresponding to each key log data, and perform deduplication processing on the replacement result. Based on the deduplicated replacement result, determine one or more task anomaly representation data corresponding to any of the target clusters.

7. The method according to any one of claims 1-4, characterized in that, The step of determining the anomaly repair information corresponding to each target cluster set based on the task anomaly representation data corresponding to each target cluster set includes: Obtain task prompt information; the task prompt information includes task content information and output format information; The task anomaly representation data and task prompt information corresponding to each target cluster set are respectively input into the large language model for processing to obtain the anomaly repair information corresponding to each target cluster set. Wherein, when the task content information includes output anomaly cause prompt information and output repair method prompt information, the anomaly repair information includes the anomaly cause and anomaly repair method corresponding to the task anomaly characterization data.

8. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to a task diagnostic request from a terminal device, the target task is determined based on the task identifier carried in the task diagnostic request, and the log data to be processed corresponding to the target task is determined. Obtain multiple task anomaly characterization data stored in the storage space; The log data to be processed is matched with the multiple task anomaly representation data respectively to obtain the matching results; If the matching result indicates that the log data to be processed matches the target task anomaly representation data among the multiple task anomaly representation data, then anomaly repair information associated with the target task anomaly representation data is determined from the storage space. The anomaly repair information associated with the anomaly representation data of the target task is returned to the terminal device.

9. A data processing apparatus, characterized in that, The device includes: The extraction unit is used to extract multiple key log data from the log data recorded by the task node for task execution. Each key log data includes keywords used to indicate task anomalies. The processing unit is used to perform clustering processing on the multiple key log data to obtain multiple target cluster sets, and to determine the task anomaly representation data corresponding to each target cluster set in the multiple target cluster sets; The processing unit is further configured to determine the anomaly repair information corresponding to each target cluster based on the task anomaly representation data corresponding to each target cluster; The storage unit is used to associate and store the task anomaly representation data and anomaly repair information corresponding to each target cluster set into the storage space, so that when the log data to be processed matches the target task anomaly representation data in the storage space, the anomaly repair information associated with the target task anomaly representation data in the storage space is output.

10. A computer device, characterized in that, The computer device includes: A processor is a tool for implementing computer programs. A computer-readable storage medium storing a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-8.

Images