Data processing method, device, apparatus, and computer-readable storage medium

Abstract

Embodiments of the present invention disclose a data processing method, apparatus, device, and computer-readable storage medium. The method includes: obtaining a log data set; matching the log data set with a preset log template library to obtain a matching degree; and determining a log clustering result based on the matching degree. By constructing a log template library to aggregate large-scale data and obtain a matching degree, the accuracy and flexibility of the log clustering results are improved. After receiving an alarm message, the fault prediction information corresponding to the log clustering result is determined based on the log clustering result and a preset multi-label classification model. This eliminates the need for operations and maintenance personnel to perform empirical problem estimation based on their own technical and business experience, thereby improving the efficiency of fault location.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Due to the massive scale of modern network systems, the ever-increasing number of applications and systems, the rapid development of high-end computing systems, and the massive demand for storage services, large-scale equipment is often deployed in hardware communication networks to ensure the stable operation of the entire system. These typically involve large-scale clusters of hundreds or even thousands of processors. Complex systems generate massive amounts of logs during operation. In system operations and business scenarios, when problems occur, operators typically need to review error logs to locate the cause.

[0003] In the existing technology, key fields in the log text are extracted through manually designed rules, and regular expressions are designed based on the keywords of each field to match the log text to obtain the log information that the operation and maintenance personnel need to pay attention to. The operation and maintenance personnel analyze the log information and combine their own technical and business experience to locate the final cause of the failure.

[0004] However, the fault location method relies on the technical and business knowledge of the operation and maintenance personnel. When an error occurs, filtering log text and locating the cause by setting conditions is very time-consuming, which is not conducive to problem solving and system recovery, and reduces the efficiency of fault location. Summary of the Invention

[0005] The embodiments of the present invention provide a data processing method, apparatus, device, and computer-readable storage medium, which improve the efficiency of fault location.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a data processing method, comprising: obtaining a log data set; matching the log data set with a preset log template library to obtain a matching degree; determining a log clustering result based on the matching degree; and after receiving an alarm message, determining fault prediction information corresponding to the log clustering result based on the log clustering result and a preset multi-label classification model.

[0008] In a second aspect, an embodiment of the present invention provides a data processing device, comprising: an acquisition module for acquiring a log data set; a matching module for matching the log data set with a preset log template library to obtain a matching degree; a clustering module for determining a log clustering result based on the matching degree; and a prediction module for determining, after receiving an alarm message, fault prediction information corresponding to the log clustering result based on the log clustering result and a preset multi-label classification model.

[0009] In a third aspect, an embodiment of the present invention provides a data processing device, comprising: a memory for storing an executable computer program; and a processor for implementing the above-mentioned data processing method when executing the executable computer program stored in the memory.

[0010] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program for implementing the above-mentioned data processing method when executed by a processor.

[0011] The embodiments of the present invention provide a data processing method, apparatus, device and computer-readable storage medium. According to the solution provided by the embodiments of the present invention, a log data set is obtained; the log data set is matched with a preset log template library to obtain a matching degree; and a log clustering result is determined based on the matching degree. By constructing a log template library, a large-scale data is aggregated to obtain a matching degree, and the log clustering result is determined based on the size of the matching degree. If the matching degree is high, the log clustering result is directly determined. If the matching degree is low, the constructed log template library supports incremental learning, and then the log clustering result is determined, thereby improving the accuracy and flexibility of the log clustering result. After receiving the alarm information, the fault prediction information corresponding to the log clustering result is determined based on the log clustering result and the preset multi-label classification model. While collecting the log data, the log data is processed and clustered in advance. When a fault alarm occurs, the fault prediction can be quickly performed based on the log clustering result, and there is no need for operation and maintenance personnel to make empirical problem estimates based on their own technology and business, thereby improving the efficiency of fault location. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of optional steps of a data processing method provided by an embodiment of the present invention;

[0013] Figure 2 An optional schematic diagram of log data set preprocessing provided by an embodiment of the present invention;

[0014] Figure 3 A flowchart of optional steps of another data processing method provided by an embodiment of the present invention;

[0015] Figure 4An optional schematic diagram of a matching process of a log template library provided by an embodiment of the present invention;

[0016] Figure 5 A flowchart of optional steps of another data processing method provided by an embodiment of the present invention;

[0017] Figure 6 An optional framework diagram of a data processing system provided by an embodiment of the present invention;

[0018] Figure 7 An optional schematic diagram of a sequence structure provided by an embodiment of the present invention;

[0019] Figure 8 An optional schematic diagram of a template tree structure provided by an embodiment of the present invention;

[0020] Figure 9 An optional schematic diagram of another template tree structure provided by an embodiment of the present invention;

[0021] Figure 10 An optional schematic diagram of another template tree structure provided by an embodiment of the present invention;

[0022] Figure 11 An optional schematic diagram of template sequence vectorization provided by an embodiment of the present invention;

[0023] Figure 12 An optional schematic diagram of a problem root cause prediction provided by an embodiment of the present invention;

[0024] Figure 13 An optional schematic diagram of a corresponding relationship provided in an embodiment of the present invention;

[0025] Figure 14 An optional schematic diagram of dimensional conversion provided by an embodiment of the present invention;

[0026] Figure 15 An optional structural diagram of a data processing device provided by an embodiment of the present invention;

[0027] Figure 16 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. It should be understood that some of the embodiments described here are only used to explain the technical solutions of the present invention and are not used to limit the technical scope of the present invention.

[0029] In order to better understand the data processing method provided in the embodiment of the present invention, the application background is first described before introducing the technical solution of the embodiment of the present invention.

[0030] The log data in the embodiment of the present invention may also be referred to as log text, log information, etc., and the fault in the embodiment of the present invention may also be referred to as problem, exception, error, error report, alarm, warning, etc., which is not limited in the embodiment of the present invention.

[0031] The data processing method provided by the embodiment of the present invention relates to the fields of Internet technology (IT) applications, system operation and maintenance, machine learning, and deep learning. In the field of system operation and maintenance and business scenarios, when a problem occurs, the operation and maintenance personnel generally need to locate the cause of the fault by checking the error log. When the business traffic is small and the logic complexity is low, the error log is generally small when the application fails, and the operation and maintenance personnel can quickly locate the problem based on the error log. However, with the iteration of business logic, the number of dependent services connected to the system continues to increase, and the number of components introduced continues to increase. When the system fails, the magnitude of the error log increases sharply. In extreme cases, the phenomenon of "crazy error reporting" occurs. At this time, the contents of the error log will have problems of mutual burial and mutual influence. The operation and maintenance personnel will find it difficult to sort out the logic when facing the error report and will lose focus. When an error occurs, if only relying on manually written rules to locate the cause of the fault will result in a very time-consuming and reduced efficiency of fault location in actual business scenarios.

[0032] In the related art, a machine learning method or a deep learning method can be used to vectorize the log text. Among them, the machine learning method can be the term frequency inverse document frequency index (TF-IDF), which is a commonly used weighting technology for information retrieval and data mining; the deep learning method can be a related model used to generate word vectors, including but not limited to deep neural network models and word2vec (word to vector). Then, the cosine similarity is used to calculate the similarity between sequences, or a clustering algorithm is used to aggregate the log text to reduce the amount of analysis for the operation and maintenance personnel. The operation and maintenance personnel analyze the aggregated log information to locate the cause of the fault. Among them, the clustering algorithm includes but is not limited to the k-means clustering algorithm (K-means) and the density-based spatial clustering of applications with noise (DBSCAN).

[0033] However, while clustering alone can reduce the amount of analysis required by operations personnel, both machine learning and deep learning methods present challenges. Machine learning methods produce inaccurate clustering results in complex business scenarios, hindering operations personnel's analysis and judgment. Deep learning methods are not widely applicable when faced with massive amounts of log data. Furthermore, after clustering, operations personnel's business experience is still required to ultimately locate the cause of the fault, reducing fault location efficiency.

[0034] When an error occurs, the related technology clusters the log alarm information within a period of time, and summarizes the log alarm information with the same root cause into a generalized alarm that can cover the alarm content, and finally forms a generalized alarm summary of only a few or dozens of items. The specific approach still requires the operation and maintenance personnel to use technical means to extract important attributes in the log alarm information based on their experience in online problem troubleshooting, such as machine, environment, time, etc., and combine the source of the exception, alarm log information prompts, and the location of the fault (for example, the number of lines of code, interface, class), etc. to form a generalized hierarchy of alarm information. However, the operation and maintenance personnel need rich online problem troubleshooting experience to obtain the alarm log summary by generalizing the alarm log information over a period of time. The log attributes and key features selected by the operation and maintenance personnel need to be accurate enough to make these generalized hierarchies effective. In addition, new alarm problems will arise over time, and the process of updating the entire alarm generalization hierarchy is too complicated.

[0035] In system operations and business scenarios, when faced with massive amounts of log data, the primary challenge for operators is quickly and accurately filtering out suspicious log data. In real-world scenarios, some faults are difficult to detect using numerical indicators and can only be discovered through textual analysis of log content. Given the sheer volume of logs, conventional keyword screening only addresses known fault causes and is incapable of addressing unknown ones. Furthermore, other fault analysis and location methods rely on parsed log information and fail to generalize the operator's own troubleshooting experience, making it difficult to develop a comprehensive methodology and system.

[0036] The embodiment of the present invention provides a data processing method, such as Figure 1 As shown, Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention includes the following steps:

[0037] S101: Obtain a log data set.

[0038] S102: Match the log data set with a preset log template library to obtain a matching degree.

[0039] In an embodiment of the present invention, the preset log template library is a pre-set standard log template library that includes multiple log templates. The templates are used to match multiple log data items in a log data set, thereby determining the number of successfully matched log data items and the number of unmatched log data items. A matching degree is calculated based on the number of successfully matched log data items and the number of unmatched log data items. The matching degree reflects the matching index between the log data items and the log templates.

[0040] In an embodiment of the present invention, when calculating whether log data matches a log template, the similarity between the sequence corresponding to the log data and the sequence corresponding to the log template can be calculated, and whether the match is successful can be determined based on the similarity. Exemplarily, when the similarity is greater than a preset threshold, the log data is determined to be a successful match; and when the similarity is less than or equal to the preset threshold, the log data is determined to be an unsuccessful match. The preset threshold can be appropriately set by those skilled in the art based on actual needs, as long as it can effectively determine whether a match exists. Exemplarily, it can be determined by analyzing a large number of thresholds used in the calculation process of a large number of similarities.

[0041] It should be noted that when the preset log template library is initially constructed, the number of log templates it includes is 0. At this time, when the log data set is first matched with the preset log template library, no log data is successfully matched. The preset log template library needs to be updated based on the log data set, and then matching is performed based on the updated log template library to obtain the matching degree. In other words, the preset log template library can be constructed and continuously updated during the matching process, without the need for pre-training or updating with a large number of log data samples, thereby achieving flexibility and diversity in log data matching. Moreover, the preset log template library is not limited to log data generated by certain systems and can process a large amount of incremental data. The incremental data can come from different systems, thus achieving compatibility of log data generated by multiple systems. When the number of unmatched log data in the log data set is large, the preset log template library can be directly updated based on the unmatched log data, and then matching can continue based on the updated log template library. There is no need to conduct additional training on the preset log template library, thereby improving matching efficiency.

[0042] S103: Determine the log clustering result according to the matching degree.

[0043] In an embodiment of the present invention, the log clustering result is determined based on the degree of matching. If the degree of matching is high, it indicates that the preset log template library is suitable for the current matching calculation process and no update is required. The successfully matched log data is directly used as the log clustering result. If the degree of matching is low, it indicates that the preset log template library is not suitable for the current matching calculation process and needs to be updated. The constructed log template library supports incremental learning to determine the log clustering result, avoiding manual intervention and improving the accuracy and flexibility of the log clustering result.

[0044] In an embodiment of the present invention, tens of thousands of log data are losslessly compressed and aggregated into dozens of templates through log clustering, and each template represents a type of log data. The log clustering result includes log templates belonging to a variety of different types. While collecting the log data of the system, the log data is summarized and abstracted in advance. When an abnormality occurs (i.e., an alarm message is received), the operation and maintenance personnel can quickly grasp the full picture of the log data, and the key error message is not easily ignored. The operation and maintenance personnel can preliminarily locate the fault through the association between the aggregated log template and the log data, thereby improving the efficiency of fault location.

[0045] S104: After receiving the alarm information, determine the fault prediction information corresponding to the log clustering result according to the log clustering result and a preset multi-label classification model.

[0046] The data processing method provided by the embodiment of the present invention is applied to the aggregation process of daily log data, and is used to provide the clustering results of log data to operation and maintenance personnel in real time, so that the operation and maintenance personnel can quickly grasp the overall picture of the log data. After receiving the alarm information or when receiving the alarm information, the prediction function is started to reduce resource consumption. When fault prediction is performed based on the log clustering results, the log clustering results are input into a preset multi-label classification model to obtain fault prediction information corresponding to the log clustering results. The fault prediction information assists the operation and maintenance personnel in reference, thereby improving the efficiency of fault location.

[0047] In an embodiment of the present invention, a preset multi-label classification model is used to predict fault information. It is trained based on the experience of operation and maintenance personnel in solving problems. The embodiment of the present invention does not limit the specific structure of the preset multi-label classification model, as long as it can predict fault information based on the log clustering results. The preset multi-label classification model can be a neural network (NN), including but not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), feed forward neural networks (FNN), fully connected neural networks (Full Connect Neural Network), etc.

[0048] According to the solution provided by the embodiment of the present invention, a log data set is obtained; the log data set is matched with a preset log template library to obtain a matching degree; and the log clustering result is determined based on the matching degree. By constructing a log template library to perform aggregation operations on large-scale data, a matching degree is obtained, and the log clustering result is determined based on the size of the matching degree. If the matching degree is high, the log clustering result is directly determined. If the matching degree is low, the constructed log template library supports incremental learning, and then the log clustering result is determined, thereby improving the accuracy and flexibility of the log clustering result. After receiving the alarm information, the fault prediction information corresponding to the log clustering result is determined based on the log clustering result and the preset multi-label classification model. While collecting the log data, the log data is processed and clustered in advance. When a fault alarm occurs, a fault prediction can be quickly made based on the log clustering result, and there is no need for operation and maintenance personnel to make empirical problem estimates based on their own technology and business, thereby improving the efficiency of fault location.

[0049] In some embodiments, as Figure 2 As shown, Figure 2 This is an optional schematic diagram of a log dataset preprocessing method provided by an embodiment of the present invention. S101 described above can also be implemented in the following manner: obtaining an original log dataset; regularizing the original log dataset to obtain a target log dataset; and deduplicating the target log dataset to obtain a log dataset.

[0050] In an embodiment of the present invention, the collected original log data often contains a lot of invalid information, which will interfere with the subsequent matching and updating of log templates, and will affect the calculation of subsequent log template vectorization. Therefore, the embodiment of the present invention uses some regularization processing to avoid the above situation. In an embodiment of the present invention, the log date, timestamp information, digital Internet Protocol (IP) class, and port class in the original log data are all information that disturbs the aggregation of the original log data. When the original log data is regularized, the above interference information is expressed in another standardized form. For example, Figure 2 As shown in the figure, different port information is normalized into "port" instead of displaying the specific port; different IP network segment information is normalized into "IP" instead of displaying the specific network segment; different timestamp information is normalized into "time" instead of displaying the specific time. This reduces the interference of irrelevant information and improves the accuracy of log data matching results.

[0051] In the embodiment of the present invention, when facing large-scale original log data, after normalizing the log data, it is necessary to perform deduplication operation on some original log data that have no substantial difference. For example, Figure 2 As shown, for two or more original log data with different port information (or different IP network segment information, or different timestamp information), after regularization, the corresponding information is the same, and they essentially belong to one log data. The duplicate log data is deduplicated to reduce the amount of data. A HashSet can be used for deduplication, and a HashSet is a set without duplicate elements; a Bloom Filter can also be used for deduplication. The embodiment of the present invention does not limit this, as long as it can achieve deduplication of the original log data. It should be noted that in the face of massive data, the storage space consumed by the HashSet is relatively large, and the read and write efficiency is also average. Therefore, the embodiment of the present invention can use a Bloom filter to deduplicate the original log data, thereby reducing the amount of data calculation.

[0052] In the embodiment of the present invention, a regularization unit is used to perform regularization processing on the current log data to obtain target log data, and the target log data is deduplicated to obtain log data ( Figure 2 In the above example, the processed data is used to represent the log data), which reduces the interference of irrelevant information, reduces the amount of data calculation, and improves the data processing efficiency.

[0053] In some embodiments, the above S102 may further include S1021-S1023. Figure 3 As shown, Figure 3A flowchart of optional steps of another data processing method provided by an embodiment of the present invention.

[0054] S1021. Segment the multiple log data in the log data set to obtain multiple word frequency information corresponding to the multiple log data.

[0055] S1022. Sort the multiple log data according to a preset word frequency library based on the multiple word frequency information to obtain multiple sequences corresponding to the multiple log data.

[0056] S1023: Matching is performed based on the multiple sequences and the multiple template sequences corresponding to the preset log template library to obtain a matching degree.

[0057] In an embodiment of the present invention, for each log data, when matching the log data, the log data is segmented, and the words or keywords in the log data are extracted to obtain the frequency information of the occurrence of multiple words (i.e., word frequency information). The multiple word frequency information of the log data are arranged in descending order according to a preset word frequency library to obtain the sequence corresponding to the log data.

[0058] In an embodiment of the present invention, the construction and update process of the preset word frequency library is similar to that of the preset log template library. It can be constructed and continuously updated during the word segmentation process of the log data. It does not require advance training or updating through a large number of word samples, thereby realizing the word flexibility and word diversity of the word frequency library.

[0059] In an embodiment of the present invention, the system log data collected during system operation and maintenance is usually the tasks completed by some programs and the status of the system printed out by the code during the program operation. Data belonging to unstructured text cannot be learned from the features and patterns of these unstructured text log data by using natural language processing (NLP) methods or statistical methods alone. There are many repeated log templates and patterns in these massive log data. Therefore, an embodiment of the present invention uses a template sequence to represent the log template, and then continuously updates it to form a preset log template library. The template sequence can be understood as a combination of frequently occurring words.

[0060] In this embodiment of the present invention, the sequence corresponding to the log data is obtained by arranging multiple word frequency information in descending order, and is essentially a combination of multiple words. Therefore, by calculating the word matching degree between the sequence corresponding to the log data and multiple template sequences corresponding to multiple log templates, it is possible to determine whether the log data successfully matches the log templates. Similarly, after calculating multiple log data, the matching degree can be obtained.

[0061] In the embodiment of the present invention, word frequency information is obtained by segmenting log data, sorting the information according to a preset word frequency library, obtaining a sequence, and then matching the sequence with multiple template sequences, thereby improving matching accuracy.

[0062] In some embodiments, the above Figure 3 The multiple template sequences corresponding to the preset log template library in S1023 can be determined in the following manner: reconstructing the root node and child nodes in the template tree structure based on the preset log template library; determining multiple paths of the template tree based on the root node and child nodes in the template tree structure; and determining multiple template sequences corresponding to the preset log template library based on the multiple paths of the template tree.

[0063] In the embodiment of the present invention, the template matching function and some post-processing steps of the log matching results are described, such as Figure 4 As shown, Figure 4 An optional schematic diagram of a log template library matching process provided by an embodiment of the present invention. Figure 4 In the example above, a preset log template library (including 300 log templates) is used to obtain multiple required log templates. Then, the log templates are loaded and a log template tree is rebuilt to match the log data. Log templates are used to store log data, so they also need to be rebuilt to obtain a template tree structure. The template tree structure includes a root node and child nodes, which can be understood as representing the relationship between individual log data. Based on the root node and child nodes in the template tree structure, multiple paths in the template tree can be calculated, and these multiple paths in the template tree are used as multiple template sequences.

[0064] In the embodiment of the present invention, the template matching function is used to match the log data set (Date) with multiple log templates (300 log templates) in the log template library to obtain the matching degree, and to determine whether the current matching degree is greater than the preset matching degree ( Figure 4 95% is used as an example) and is represented by Figure 4 The post-processing module in the post-processing step is used. When the matching degree is greater than 95%, the log matching result is output; when the matching degree is less than or equal to 95%, the update operation of the preset log template library is executed ( Figure 4 After the above-mentioned regularization and deduplication processing of the original log data, the log data is obtained in order to reduce the amount of data calculation. However, the log clustering results need to be displayed to the operation and maintenance personnel so that the operation and maintenance personnel can quickly grasp the overall picture of the log data. Therefore, after obtaining the log matching results, Figure 4The post-processing module also performs subsequent operations on the log matching results. Log matching results are essentially the result of aggregating multiple log data belonging to the same log template. The log data information in the log matching results (for example, log date, timestamp information, digitized IP class, port class, etc.) is transformed to achieve post-processing of the log matching results and output the final log clustering results for easy display and storage to operation and maintenance personnel.

[0065] In some embodiments, in the above Figure 1 S103 may include S1031-S1033. Figure 5 As shown, Figure 5 A flowchart of optional steps of another data processing method provided by an embodiment of the present invention.

[0066] S1031: If the matching degree is greater than the preset matching degree, the successfully matched log data is used as the log clustering result.

[0067] S1032: If the matching degree is less than or equal to the preset matching degree, the preset log template library is updated according to the unmatched log data to obtain an updated log template library.

[0068] S1033: Match the log data set with the updated log template library to obtain a log clustering result.

[0069] The data processing method provided in the embodiments of the present invention can be executed by a data processing system (i.e., a data processing device), which can also be referred to as a fault location system or a problem location system. The data processing system includes a raw data processing module (also referred to as a data processing module), a template matching module (i.e., a matching module), a template update and maintenance module (also referred to as an update module), and a problem root cause prediction module (i.e., a prediction module). The raw data processing module includes a regularization unit and a deduplication unit.

[0070] For example, Figure 6 As shown, based on the above Figure 4 , Figure 6 An optional framework diagram of a data processing system provided by an embodiment of the present invention. (1) The data processing module pre-processes the original log data input by the log collection system, standardizes the log data through the regularization unit, and then performs deduplication operation through the deduplication unit to reduce the overall data calculation amount. (2) The processed log data (i.e., log data set, Figure 6 ) and the current log template library ( Figure 6In the example, the current log template library includes 300 log templates (300 templates) for matching. The current log template library represents the latest log template library (which can be a preset log template library or an updated log template library). If the overall matching degree is greater than the preset matching degree ( Figure 6 The remaining unmatched log data is classified into a separate category, or the remaining unmatched log data is deleted, and the final log clustering result is output. The specific implementation and technical effects achieved in this step can be referred to above. Figure 4 (3) If the matching degree between the log data and the log template library is less than or equal to the preset matching degree ( Figure 6 (shown in the example of ≤95%), the unmatched log data is incrementally learned, i.e., log template calculation, and then the log template library is updated and maintained. (4) After the log template library is updated, the log data is matched with the updated log template library again, and the final clustering result is output. (5) The candidate problem templates in the system during the current time period (i.e., log clustering results, which include log templates belonging to various types) are vectorized, and the root cause of the problem is predicted (i.e., fault information is predicted) based on the vectorized candidate problem templates.

[0071] In an embodiment of the present invention, a log template library is constructed to aggregate the massive amount of log data in the field of system operation and maintenance, thereby improving the efficiency of log data aggregation. When dealing with newly added log data, the log template calculation step supports incremental learning, avoiding manual intervention and improving the accuracy of log data clustering. The flexible and diverse clustering implementation is compatible with multiple system framework languages, increasing the applicability of the log template library to various scenarios.

[0072] In some embodiments, the above Figure 5 In S1032, updating the preset log template library based on the unmatched log data to obtain an updated log template library can also be achieved in the following manner: performing word segmentation on the unmatched log data to obtain word frequency information corresponding to the unmatched log data; updating the preset word frequency library based on the word frequency information corresponding to the unmatched log data to obtain an updated word frequency library; sorting the unmatched log data according to the updated word frequency library based on the word frequency information corresponding to the unmatched log data to obtain a sequence corresponding to the unmatched log data; merging the sequence corresponding to the unmatched log data with multiple template sequences corresponding to the preset log template library to obtain an updated log template library.

[0073] In an embodiment of the present invention, for the input log data, if the matching degree is less than or equal to the preset matching degree, it is necessary to perform incremental learning of the log template based on the unmatched log data, and the process is as follows: perform word segmentation processing on the unmatched log data to calculate the frequency of occurrence of words, and obtain the word frequency information corresponding to the unmatched log data; and update the current word frequency dictionary (i.e., the preset word frequency library) based on the word frequency information, WFRE{}→NEW_WFRE{}, WFRE{} represents the preset word frequency library, and NEW_WFRE{} represents the updated word frequency library; the word segmentation processing in this step can refer to the relevant description of S1021 above. The unmatched log data is processed according to the word frequency information, and is arranged in descending order according to the updated word frequency library NEW_WFRE{}, and all the arranged sequences are used as the newly added template sequences (i.e., the sequences corresponding to the unmatched log data). The sorting in this step can refer to the relevant description of S1022 above. The newly added template sequence is merged with the template sequence Patterns{} corresponding to the existing log template library (that is, multiple template sequences corresponding to the preset log template library), and the log template calculation process is executed to rebuild the template tree structure. Finally, all paths of the new template tree are calculated. After the paths are saved, the template sequence NEW_Patterns{} corresponding to the new log template library is obtained, thereby obtaining the updated log template library.

[0074] In the embodiment of the present invention, the calculation process of the log template is introduced, that is, the sequences arranged in descending order (including the newly added template sequence and multiple template sequences corresponding to the preset log template library) are merged by constructing a template tree structure. The merging of two sequences is used as an example for explanation. Figure 7 As shown, Figure 7 An optional schematic diagram of a sequence structure provided by an embodiment of the present invention. (1) Create the root node of the template tree, and then Figure 7 The sequence S1 and sequence S2 shown in the figure are merged, wherein sequence S1 includes nodes W11, W12, W13, W14 and W15, and sequence S2 includes nodes W11, W12, W13, W24 and W25. (2) When processing sequence S1, the nodes in sequence S1 are continuously inserted into the template tree. When processing sequence S2, if the nodes are the same, they are skipped directly. If different nodes are encountered, a new path is opened in the template tree. The template tree structure is as follows: Figure 8 As shown, Figure 8 An optional schematic diagram of a template tree structure provided by an embodiment of the present invention. Figure 8There are two paths in it, one is node W11→node W12→node W13→node W14→node W15, and the other is node W11→node W12→node W13→node W24→node W25. (3) Similarly, the same method is used for the remaining sequence merging, and the current template tree is continuously improved. The complete template tree can be obtained as follows Figure 9 As shown, Figure 9 An optional schematic diagram of another template tree structure provided by an embodiment of the present invention, thereby completing the template tree construction process. Figure 9 Where ROOT represents the root node, and the paths include node W11→node W12→node W13→node W14→node W15, node W11→node W12→node W13→node W24→node W25, node W31→node W32, node W31→node W33→node W34→node W35→node W36, node W31→node W33→node W34→node W35→node W37, node W31→node W33→node W34→node W35→node W38. (4) After merging all sequences, the final template tree structure is obtained, and the paths passing through all child nodes represent different log templates. In addition, the embodiment of the present invention also provides a pruning mechanism. If the child node tree of the current layer is greater than or equal to the set threshold, it is pruned to prevent template tree redundancy. Figure 10 As shown, Figure 10 An optional schematic diagram of another template tree structure provided in an embodiment of the present invention. Figure 10 Nodes W36, W37, and W38 are located in the same layer, and the number of child node trees is equal to the set threshold ( Figure 10 Taking 3 as an example), it is necessary to perform a pruning (CUT) operation on the nodes W36, W37, and W38 in the layer, that is, there is no such layer in the template tree, thereby preventing the template tree from being redundant. Figure 10 The middle paths include node W11 → node W12 → node W13 → node W14 → node W15, node W11 → node W12 → node W13 → node W24 → node W25, node W31 → node W32, node W31 → node W33 → node W34 → node W35. (5) Calculate all paths in the current template tree and save the paths to obtain the template sequence NEW_Patterns{} corresponding to the new log template library, realizing incremental learning of log templates.

[0075] It should be noted that the above-mentioned log template calculation forms the basis for updating and maintaining the entire log template library. When the preset log template library is initially constructed, the number of log templates it includes is zero. At this point, when the log dataset is first matched against the preset log template library, no log data is successfully matched. Before the matching process, the preset log template library must be initially constructed based on the log dataset. This initial construction process is essentially an incremental learning process for log templates. The initial construction process for the preset log template library is as follows: the log dataset is segmented to calculate the frequency of word occurrences, resulting in a word frequency dictionary WFRE{} (i.e., the preset word frequency dictionary) and word frequency information corresponding to each log data point. Based on this word frequency information, each log data point in the log dataset is processed again, and the log data points are sorted in descending order according to the word frequency dictionary WFRE{}. All sorted sequences are used as candidate log templates. During the above-mentioned log template calculation process, a template tree structure is obtained. All paths in the current template tree are calculated, and the paths are saved to obtain the template sequence Patterns{} corresponding to the required log template library (i.e., the template sequence corresponding to the preset log template library).

[0076] In this embodiment of the present invention, when constructing a log template library, log templates can be automatically extracted from system log data without relying on any prior knowledge. Log templates can represent subtypes of detailed information fields in system log data, improving the accuracy of the log template library. Furthermore, incremental log template learning eliminates the need for manual control of log template library updates, increasing the applicability of the log template library to various scenarios.

[0077] In some embodiments, after a preset period of time, or after the above Figure 5 After obtaining the updated log template library in S1032, the data processing method further includes S201 and S202.

[0078] S201. If the number of multiple template sequences corresponding to the log template library is greater than a preset number, the support of each template sequence is determined according to the log hit number and log hit time of each template sequence, thereby obtaining multiple support degrees; the log template library is a preset log template library or an updated log template library.

[0079] S202 : Screen multiple template sequences corresponding to the log template library according to multiple support degrees to obtain a maintained log template library. The maintained log template library is used for a matching process of a next log data set.

[0080] In some embodiments, the above S202 can also be implemented in the following manner. Template sequences with a support greater than a preset support among multiple supports are used as multiple candidate template sequences; if the number of multiple candidate template sequences is less than or equal to the preset number, a log template library after maintenance is determined based on the multiple candidate template sequences; if the number of multiple candidate template sequences is greater than the preset number, vector similarities between template sequences with a support less than or equal to the preset support and the multiple candidate template sequences are calculated to obtain multiple vector similarities; based on the multiple vector similarities, multiple target template sequences are screened out from the multiple candidate template sequences; the number of multiple target template sequences is less than or equal to the preset number; and a log template library after maintenance is determined based on the multiple target template sequences.

[0081] In an embodiment of the present invention, as time goes by, the number of templates in the log template library will increase, and too many log templates will lose the meaning of aggregating log data. Therefore, in an embodiment of the present invention, the data processing system is also provided with a log template maintenance engine to avoid the explosive growth of log templates. Before introducing the maintenance of the log template library, the support of the template sequence used in the maintenance process is first introduced. Each subsequence used as a log template has its own "support". Only subsequences with a sufficiently large "support" can be added to the log template library. The calculation of the "support" of the entire sequence is based on the different word combinations in the current log data. Therefore, the word frequency is used as the basis for calculating the "support". The support of the log template, that is, the support of the template sequence, is calculated based on the number of log hits of the log template and the most recent hit time of the log template.

[0082] γ=lnβ*(log2α+1) (1)

[0083] In the above formula (1), α represents the number of log hits, β represents the most recent hit time of the log template (timestamp is a floating point number of seconds), and γ represents the support of the log template.

[0084] In an embodiment of the present invention, the process of maintaining a log template library involves clearing out expired or "unimportant" templates. An upper limit N (i.e., a preset number) is set for the total number of log templates. Multiple support levels are compared with the preset support level, and template sequences with support levels less than or equal to the preset support level are deleted. If the number of template sequences obtained after deletion is less than or equal to the preset number N, a maintained log template library is obtained. If the number of template sequences obtained after deletion is greater than the preset number N, the following steps are performed.

[0085] In the embodiment of the present invention, when calculating the vector similarity, it is necessary to first serialize the log template to obtain the sentence vector of the template sequence that is less than or equal to the preset support, and the sentence vector of the candidate template sequence (i.e., the template sequence that is greater than the preset support). Here, a pre-trained model is used to serialize the log template to extract the log template sentence vector. Figure 11 As shown, Figure 11 An optional schematic diagram of template sequence vectorization provided by an embodiment of the present invention, Figure 11 The log template text represents a template sequence, which includes multiple subsequences, such as Flig, Time, and Base_set. The pre-trained model can be an encoder. The log template text is input into the pre-trained model to obtain a vectorized representation of the template sequence, thereby completing the vectorization process of the template sequence.

[0086] In an embodiment of the present invention, the specific structure of the preset training model can refer to the encoding and decoding structure of the Transformer or the encoding and decoding structure of the Deformable DETR (Deformable Transformers for end-to-end object detection) network, which will not be repeated here.

[0087] In an embodiment of the present invention, the sentence vector of the log template with a support less than or equal to the preset support is compared with the sentence vector of the candidate template sequence in the log template library using the cosine distance to calculate the vector similarity between the two.

[0088]

[0089] In the above formula (2), θ represents the threshold for removing log templates. Similarity represents vector similarity, and A and B represent two vectors, namely, the sentence vector of the template sequence with a support less than or equal to the preset support and the sentence vector of the candidate template sequence.

[0090] It should be noted that a distance metric is usually used to calculate the similarity between two vectors. Common distance metrics include, but are not limited to, cosine similarity, Euclidean distance, Manhattan distance, Pearson correlation coefficient (PC), Spearman Rank Correlation (SRC), Jaccard distance, SimHash+Hamming distance, etc. Here, only the calculation of the cosine distance in formula (2) above is used as an example to illustrate.

[0091] Exemplarily, after obtaining multiple vector similarities, a preset number N of target template sequences are selected from multiple candidate template sequences in descending order of vector similarity; or, a preset number N of target template sequences are selected from multiple candidate template sequences in descending order of vector similarity. The embodiment of the present invention does not limit the specific screening method. Template sequences with a support less than or equal to the preset support have been deleted, indicating that they do not meet the criteria for adding to the log template library. Template sequences with a high vector similarity to log templates with lower support (less than or equal to the preset support) also do not meet the criteria for adding to the log template library. Therefore, the above screening process is to delete template sequences with a high vector similarity to log templates with lower support, which can also be understood as including template sequences with a low vector similarity to log templates with lower support.

[0092] In the embodiment of the present invention, after obtaining multiple vector similarities, the vector similarities are compared, and the top vectors are deleted after sorting according to the vector similarities. k Log templates are added until the total number of log template libraries is ≤ N. The relevant parameters required by the above log template maintenance engine include α, β, γ, N, and θ.

[0093] In an embodiment of the present invention, the template update and maintenance function can automatically extract log templates from system log data. The log template can represent the pattern of the detailed information field in the system log message, thereby improving the accuracy of the log template library. The log template supports incremental learning. For the update and maintenance of the log template, a log template maintenance engine is proposed. The support of the template sequence of the log template is calculated by the number of hits in the log and the time of the most recent hit, and the template sequence is vectorized in combination with the pre-trained model. The log template library is maintained according to the vector similarity, avoiding the unlimited growth of the log template library. On the basis of the continuous update of the log template library, the timeliness of the log template library under the long-term operation of the current data processing system is guaranteed. The update and maintenance of the log template library is one of the important parts of the entire system, ensuring the closed-loop operation from log aggregation to the update and maintenance of the entire log template library, thereby improving the accuracy of subsequent log clustering results and the accuracy of subsequent problem root cause predictions.

[0094] In some embodiments, the preset multi-label classification model is a single-level model; Figure 1 S104 can also be implemented in the following manner: embedding and encoding the log clustering results to obtain features to be classified; inputting the features to be classified into a preset multi-label classification model to obtain potential problem labels and problem types; and providing the fault prediction information as a multi-level prediction result, the fault prediction information includes potential problem labels and problem types.

[0095] In an embodiment of the present invention, in current actual business scenarios, when locating a fault, both the problem type (the layer where the problem occurs, for example, the network layer, virtual layer, physical layer, and software layer) and the potential problem label (the type of problem, for example, Label-1, Label-2, and Label-3) are required, and there is a corresponding relationship between the two. This is actually a hierarchical multi-label classification, but it is different from general hierarchical multi-label classification. Here, the model prediction results need to ensure that the "potential problem label" and the "problem type" can fully correspond, while also ensuring the accuracy of the predicted "problem type." In some embodiments, a multi-level model composed of multiple sub-networks can be used to achieve the prediction effect of multi-label classification. The first sub-network and the second sub-network of the multi-level model are used to predict the "potential problem label" based on the log clustering results, and the second sub-network is used to predict the "problem type" based on the "potential problem label" output by the first sub-network, thereby obtaining the potential problem label and problem type corresponding to the log clustering results. The potential problem label and problem type predicted by the multi-level model composed of multiple sub-networks have a corresponding relationship, but it is easy for the potential problem label and problem type to be wrong at the same time, and the complexity of the multi-level model is high and the training effect is poor. Therefore, in the embodiment of the present invention, the data processing system sets a single-level model, that is, a single-level multi-label classification model is used to achieve a multi-level prediction effect, which can reduce the complexity of the model. The preset multi-label classification model uses a pre-trained model with a small number of layers combined with a lightweight convolutional neural network to embed the log clustering results to obtain the features to be classified; the features to be classified are input into the single-level model, and the multi-level prediction results, i.e., the potential problem label and problem type, are output.

[0096] In the embodiment of the present invention, through the continuous updating and maintenance of the log template library, and the continuous matching of log data, the massive log data has been compressed into some specific log templates, which reduces the amount of data analysis for operation and maintenance personnel and improves data processing efficiency. However, for the fault location caused by the problem, it is still necessary for the operation and maintenance personnel to analyze the log template content and the log text content to make a judgment. Therefore, based on the log clustering function, the data processing system in the embodiment of the present invention is also provided with a problem root cause prediction function. At the level of the summarized and generalized logs, an iteratively optimized problem location mechanism is designed, which provides an online learning and updated multi-label classification model. After the problem occurs, the labels of potential problem log templates are predicted in advance, and the operation and maintenance personnel's own technical and business-level experience can be generalized to quickly locate the cause of the fault. In this way, the fault information of the log clustering results can be predicted, and the operation and maintenance personnel can be assisted in locating the fault location, thereby improving the efficiency of fault location.

[0097] In some embodiments, the data processing method further includes a model training step, including S301 and S302.

[0098] S301: Obtain the true value of fault information corresponding to the log clustering result.

[0099] S302. Based on the true value of the fault information and the fault prediction information, a preset loss function is used to update the preset multi-label classification model to obtain an updated multi-label classification model. The updated multi-label classification model is used to determine the fault prediction information of the next log clustering result.

[0100] In the embodiment of the present invention, Figure 12 As shown, Figure 12 This is an optional schematic diagram of a problem root cause prediction method provided by an embodiment of the present invention. When operations personnel receive a warning message (WARNING) from the log collection system, they can use the predicted labels provided by the problem location function to perform root cause analysis. The predicted labels include the problem type corresponding to the current log template and the potential problem label. If the predicted labels contain errors or new template label pairs (corresponding relationships between candidate templates, corrected labels for potential problems, and corrected labels for problem types) are available, operations personnel can perform online label corrections for the predicted labels (i.e., the true value of the fault information corresponding to the log clustering results). Figure 12 The predicted label is obtained by inputting the N log templates (N Templates) included in the log template library into the multi-label classification model. Figure 12 The 010 010110 in the example represents the embedding vector corresponding to the log template library. The log clustering result is essentially a variety of log templates. Label correction corrects potential problem labels and problem types, thereby updating the multi-label classification model and storing the template label pairs in the problem label library. Figure 12 The problem label library shows four candidate templates (pattern1, pattern2, pattern3, pattern4), among which the potential problem labels corresponding to pattern1 are Label-1 and Label-2, and the corresponding problem type is the network layer; the potential problem labels corresponding to pattern2 are Label-2 and Label-3, and the corresponding problem type is the virtual layer; the potential problem label corresponding to pattern3 is Label-3, and the corresponding problem type is the physical layer; the potential problem label corresponding to pattern4 is Label-4, and the corresponding problem type is the software layer.

[0101] For example, when a problem alarm message is generated, the operation and maintenance personnel mark the log template where the problem occurs based on the problem analysis results in the actual business scenario, or correct the label based on the existing prediction results (i.e., fault prediction information), and store the template label pair in the problem label library; and update the multi-label classification model online.

[0102] In an embodiment of the present invention, a loss value is obtained based on the true value of the fault information, the fault prediction information and the preset loss function; the preset multi-label classification model is continuously trained based on the loss value until the training termination condition is reached, for example, the number of training times reaches a preset number of times, or the loss value reaches a preset threshold, etc., to obtain an updated multi-label classification model.

[0103] It should be noted that the preset loss function (loss function) can be a loss function appropriately set by those skilled in the art according to actual conditions, and can be any of the following: intersection-over-union ratio (DiceLoss), smooth SmoothL1 loss function, logarithmic loss function (logLoss, LR), hinge loss function (hinge loss, SVM), exponential loss function (exp-loss, AdaBoost), cross-entropy loss function (cross-entropy loss, Softmax), square error loss function (quadratic loss), absolute value loss function (absolute loss) and 0-1 loss function (0-1loss), etc., which are not limited to the embodiments of the present invention.

[0104] In some embodiments, the true value of the fault information includes the true value of the potential problem label and the true value of the problem type. S302 described above may also be implemented in the following manner: The true value of the problem type and the true value of the potential problem label are respectively embedded and encoded to obtain a type vector and an initial label vector; the initial label vector is dimensionally transformed to obtain a label vector such that the dimension of the label vector is consistent with that of the type vector; and a preset loss function is used to update a preset multi-label classification model based on the type vector and label vector corresponding to the log clustering results, as well as the fault prediction information, to obtain an updated multi-label classification model.

[0105] In the embodiment of the present invention, additional steps are required to update the model to ensure the accuracy of model prediction and the requirements of online training, so as to achieve the multi-level prediction effect using a single-level multi-label classification model. Figure 13 As shown, Figure 13 An optional schematic diagram of a corresponding relationship provided for an embodiment of the present invention. Figure 13It shows that the template label pairs (correspondence between candidate templates, potential problem labels and problem types) are stored in the problem label library. In the embodiment of the present invention, the problem type is obtained by reverse deduction from the potential problem label, which can ensure that the two are completely corresponding, that is, the prediction label directly output by the multi-label classification model is the "potential problem label", and the "problem type" is obtained through post-processing of the correspondence table. The log template is input into the multi-label classification model, and the potential problem label is output. According to the correspondence table, the problem type is reversely deduced, and the potential problem label and the problem type are stored in the problem label library according to the correspondence. For a specific description, please refer to the above Figure 12 The description is not repeated here.

[0106] In the embodiment of the present invention, an additional step is required to allow the "question type" to participate in the training process of the multi-label classification model to ensure the accuracy of the "question type" and avoid the situation where both the potential question label and the question type are wrong at the same time. During the data processing process, the one-hot encoding (onehot_labels) dimension of the digital label of the "potential question label" is converted to obtain the label encoding corresponding to the "question type". The conversion process can be obtained by calculation between matrices, such as Figure 14 As shown, Figure 14 An optional schematic diagram of dimensional conversion provided by an embodiment of the present invention. Figure 14 Where M*N represents the preset matrix, and the preset matrix is ​​combined with the initial label vector ( Figure 14 Labels×M) are multiplied to obtain the label vector ( Figure 14 The dimension of the label vector is consistent with that of the type vector.

[0107] It should be noted that, as mentioned above Figure 12 and Figure 13 As shown in the figure, since the true value of the question type is unique, but the true value of the potential question label is not unique, when performing dimension conversion, the initial label vector corresponding to the true value of the potential question label is dimensionally converted so that the dimension of the obtained label vector is consistent with the dimension of the type vector.

[0108] In an embodiment of the present invention, through dimensionality conversion, both the true value of the problem type and the true value of the potential problem label are simultaneously incorporated into the training and updating processes of the multi-label classification model. This allows the multi-label classification model to output both the problem type and the potential problem label, improving the accuracy of the problem type. After the multi-label classification model is updated online, it infers and predicts the root cause of the problem in the log template library and outputs candidate templates corresponding to the potential problem label and problem type. This provides diverse reference information to operation and maintenance personnel, thereby improving the efficiency of problem location.

[0109] In some embodiments, the preset loss function is the product of the loss function corresponding to the potential problem label and a first preset coefficient, and the sum of the product of the loss function corresponding to the problem type and a second preset coefficient.

[0110] For example, the cross entropy loss function is used as the optimization objective function of the preset multi-label classification model. During the model training process, a loss function related to the "problem type" is created. The loss function formula is shown in formula (3).

[0111]

[0112] In the above formula (3), L K Represents the loss function corresponding to the problem type, L i represents the loss function corresponding to the candidate template i, N represents the number of candidate templates, y ic represents the number of candidate templates i that are question types c, p ic Candidate template i is the probability of question type c, and M represents the number of question types. K It is included in the total loss function Loss, as shown in formula (4).

[0113] Loss=α*L K +(1-α)*L P (4)

[0114] In the above formula (4), Loss represents the preset loss function, L P The loss function corresponding to the potential problem label is also a cross entropy loss function. For details, please refer to the description of formula (3) above, which will not be repeated here. α represents the adjustment coefficient (i.e., the second preset coefficient), ranging from 0 to 1, and 1-α is the first preset coefficient.

[0115] In response to the need for intelligent operation and maintenance of massive log data, an embodiment of the present invention provides a problem location method based on a log template library combined with root cause prediction. First, the massive log data is regularized and deduplicated. Then, log templates are calculated on the log data to compress the log data into a small number of log templates. Furthermore, by setting up an incremental learning and log template maintenance engine, the log template library is continuously updated and maintained, thereby achieving the purpose of log aggregation. This not only improves the accuracy of log clustering results, but also ensures the timeliness of the log template library under the long-term operation of the current system. Then, based on the log clustering results, the technical and business experience of the operation and maintenance personnel in solving problems is generalized. By combining the operation and maintenance personnel's problem-solving experience, candidate problem templates are labeled and corrected. Furthermore, a lightweight pre-trained model embedding is combined with a convolutional neural network to update the multi-label classification model online. Through additional design of the model training and prediction steps, the single-layer multi-label classification model achieves multi-layer prediction effects, establishing a problem root cause prediction function. It is not just a simple display of log clustering results. The root cause prediction function can calculate the problem labels and potential problem labels of candidate problem templates for reference by operation and maintenance personnel, assisting them in locating the cause of the fault and improving the efficiency of problem location.

[0116] In order to implement the data processing method of the embodiment of the present invention, the embodiment of the present invention also provides a data processing device, such as Figure 15 As shown, Figure 15 An optional structural diagram of a data processing device provided in an embodiment of the present invention, the data processing device 150 includes: an acquisition module 1501, used to obtain a log data set; a matching module 1502, used to match the log data set with a preset log template library to obtain a matching degree; a clustering module 1503, used to determine a log clustering result based on the matching degree; and a prediction module 1504, used to determine, after receiving an alarm message, fault prediction information corresponding to the log clustering result based on the log clustering result and a preset multi-label classification model.

[0117] In some embodiments, the matching module 1502 is also used to segment multiple log data in the log data set to obtain multiple word frequency information corresponding to the multiple log data; based on the multiple word frequency information, the multiple log data are sorted according to a preset word frequency library to obtain multiple sequences corresponding to the multiple log data; based on the multiple sequences and the multiple template sequences corresponding to the preset log template library, matching is performed to obtain a matching degree.

[0118] In some embodiments, the matching module 1502 is further used to reconstruct the root node and child nodes in the template tree structure based on the preset log template library; determine multiple paths of the template tree based on the root node and child nodes in the template tree structure; and determine multiple template sequences corresponding to the preset log template library based on the multiple paths of the template tree.

[0119] In some embodiments, the data processing device 150 further includes an update module 1505;

[0120] The clustering module 1503 is further configured to use the successfully matched log data as the log clustering result if the matching degree is greater than a preset matching degree;

[0121] An updating module 1505 is configured to update the preset log template library according to the unmatched log data to obtain an updated log template library if the matching degree is less than or equal to the preset matching degree;

[0122] The clustering module 1503 is further configured to match the log data set with the updated log template library to obtain a log clustering result.

[0123] In some embodiments, the update module 1505 is also used to segment the unmatched log data to obtain word frequency information corresponding to the unmatched log data; update the preset word frequency library according to the word frequency information corresponding to the unmatched log data to obtain an updated word frequency library; sort the unmatched log data according to the updated word frequency library according to the word frequency information corresponding to the unmatched log data to obtain a sequence corresponding to the unmatched log data; merge the sequence corresponding to the unmatched log data with multiple template sequences corresponding to the preset log template library to obtain an updated log template library.

[0124] In some embodiments, the update module 1505 is also used to determine the support of each template sequence based on the number of log hits and the log hit time of each template sequence if the number of multiple template sequences corresponding to the log template library is greater than a preset number, thereby obtaining multiple support degrees; the log template library is a preset log template library or an updated log template library; according to the multiple support degrees, the multiple template sequences corresponding to the log template library are screened to obtain a maintained log template library, and the maintained log template library is used for the matching process of the next log data set.

[0125] In some embodiments, the update module 1505 is further used to use template sequences with a support greater than a preset support among multiple supports as multiple candidate template sequences; if the number of multiple candidate template sequences is less than or equal to the preset number, then the maintained log template library is determined based on the multiple candidate template sequences; if the number of multiple candidate template sequences is greater than the preset number, then the vector similarities between the template sequences with a support less than or equal to the preset support and the multiple candidate template sequences are calculated to obtain multiple vector similarities; based on the multiple vector similarities, multiple target template sequences are screened out from the multiple candidate template sequences; the number of multiple target template sequences is less than or equal to the preset number; and the maintained log template library is determined based on the multiple target template sequences.

[0126] In some embodiments, the preset multi-label classification model is a single-level model;

[0127] The prediction module 1504 is also used to embed the log clustering results to obtain features to be classified; input the features to be classified into a preset multi-label classification model to obtain potential problem labels and problem types; the fault prediction information is a multi-level prediction result, and the fault prediction information includes potential problem labels and problem types.

[0128] In some embodiments, the data processing device 150 further includes a training module 1506;

[0129] The acquisition module 1501 is further used to obtain the true value of the fault information corresponding to the log clustering result;

[0130] The training module 1506 is used to update the preset multi-label classification model based on the true value of the fault information and the fault prediction information using a preset loss function to obtain an updated multi-label classification model. The updated multi-label classification model is used to determine the fault prediction information of the next log clustering result.

[0131] In some embodiments, the fault information truth value includes a potential problem label truth value and a problem type truth value;

[0132] The training module 1506 is also used to embed and encode the true value of the problem type and the true value of the potential problem label respectively to obtain a type vector and an initial label vector; perform dimension conversion on the initial label vector to obtain a label vector so that the dimension of the label vector is consistent with that of the type vector; and update the preset multi-label classification model using a preset loss function based on the type vector and label vector corresponding to the log clustering results, as well as the fault prediction information, to obtain an updated multi-label classification model.

[0133] In some embodiments, the preset loss function is the product of the loss function corresponding to the potential problem label and a first preset coefficient, and the sum of the product of the loss function corresponding to the problem type and a second preset coefficient.

[0134] In some embodiments, the data processing device 150 further includes a data processing module 1507;

[0135] The acquisition module 1501 is also used to obtain the original log data set;

[0136] The data processing module 1507 is used to regularize the original log data set to obtain a target log data set; and to remove duplicates from the target log data set to obtain a log data set.

[0137] It should be noted that the data processing device provided in the above embodiment only uses the division of the above program modules as an example to illustrate when performing data processing. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment are based on the same concept. The specific implementation process and beneficial effects are detailed in the method embodiment and will not be repeated here. For technical details not disclosed in the embodiment of this device, please refer to the description of the method embodiment of the present invention for understanding.

[0138] In an embodiment of the present invention, Figure 16 This is a schematic diagram of the structure of the data processing device proposed in an embodiment of the present invention, as shown in FIG. Figure 16 As shown, the device 160 provided in the embodiment of the present invention includes a processor 1601 and a memory 1602 storing an executable computer program. The processor 1601 is configured to implement the data processing method provided in the embodiment of the present invention when executing the executable computer program stored in the memory 1602. In some embodiments, the data processing device 160 may further include a communication interface 1603 and a bus 1604 for connecting the processor 1601, the memory 1602, and the communication interface 1603.

[0139] In the embodiment of the present invention, the processor 1601 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present invention does not specifically limit this.

[0140] In the embodiment of the present invention, the bus 1604 is used to connect the communication interface 1603, the processor 1601 and the memory 1602 to achieve mutual communication between these devices.

[0141] Memory 1602 is used to store executable computer programs and data. The executable computer programs include computer operating instructions. Memory 1602 may include high-speed RAM memory, or may also include non-volatile memory, such as at least two disk memories. In practical applications, the memory 1602 may be a volatile memory (such as random-access memory (RAM); or a non-volatile memory (such as read-only memory (ROM), flash memory (flash memory), a hard disk drive (HDD), or a solid-state drive (SSD); or a combination of the above types of memory, and provides the executable computer programs and data to the processor 1601.

[0142] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0143] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0144] An embodiment of the present invention provides a computer-readable storage medium storing a computer program for implementing the data processing method described in any of the above embodiments when executed by a processor.

[0145] Illustratively, the program instructions corresponding to a data processing method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to a data processing method in the storage medium are read or executed by an electronic device, the data processing method described in any of the above embodiments can be implemented.

[0146] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0147] The present invention is described with reference to implementation flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flowcharts. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which is implemented in the implementation flow diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process described in the flowchart. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Get the log dataset; Matching the log data set with a preset log template library to obtain a matching degree; Determining a log clustering result according to the matching degree; After receiving the alarm information, determine the fault prediction information corresponding to the log clustering result based on the log clustering result and the preset multi-label classification model; The preset multi-label classification model is a single-layer model; Determining fault prediction information corresponding to the log clustering result based on the log clustering result and a preset multi-label classification model includes: Embedding the log clustering results to obtain features to be classified; Inputting the features to be classified into the preset multi-label classification model to obtain potential problem labels and problem types; the fault prediction information is a multi-level prediction result, and the fault prediction information includes the potential problem labels and the problem types; The method further comprises: Obtaining the true value of the fault information corresponding to the log clustering result; According to the true value of the fault information and the fault prediction information, the preset multi-label classification model is updated using a preset loss function to obtain an updated multi-label classification model, and the updated multi-label classification model is used to determine the fault prediction information of the next log clustering result; The fault information truth value includes the potential problem label truth value and the problem type truth value; The method of updating the preset multi-label classification model based on the true value of the fault information and the fault prediction information using a preset loss function to obtain an updated multi-label classification model includes: Embedding the true value of the question type and the true value of the potential question label respectively to obtain a type vector and an initial label vector; Performing dimension conversion on the initial label vector to obtain a label vector, so that the dimension of the label vector is consistent with that of the type vector; According to the type vector and label vector corresponding to the log clustering result, and the fault prediction information, the preset multi-label classification model is updated using the preset loss function to obtain the updated multi-label classification model; The preset loss function is the product of the loss function corresponding to the potential problem label and a first preset coefficient, and the sum of the product of the loss function corresponding to the problem type and a second preset coefficient.

2. The method according to claim 1, characterized in that The matching of the log data set with a preset log template library to obtain a matching degree includes: Performing word segmentation on multiple log data in the log data set to obtain multiple word frequency information corresponding to the multiple log data; According to the plurality of word frequency information, the plurality of log data are sorted according to a preset word frequency library to obtain a plurality of sequences corresponding to the plurality of log data; Matching is performed based on the multiple sequences and the multiple template sequences corresponding to the preset log template library to obtain the matching degree.

3. The method according to claim 2, characterized in that Before obtaining the matching degree by matching the multiple sequences and the multiple template sequences corresponding to the preset log template library, the method further includes: Rebuilding the root node and child nodes in the template tree structure according to the preset log template library; Determining multiple paths of the template tree according to the root node and child nodes in the template tree structure; The multiple template sequences corresponding to the preset log template library are determined according to the multiple paths of the template tree.

4. The method according to claim 1, wherein Determining the log clustering result according to the matching degree includes: If the matching degree is greater than the preset matching degree, the successfully matched log data is used as the log clustering result; If the matching degree is less than or equal to the preset matching degree, updating the preset log template library according to the unmatched log data to obtain an updated log template library; The log data set is matched with the updated log template library to obtain the log clustering result.

5. The method according to claim 4, characterized in that The updating of the preset log template library according to the unmatched log data to obtain an updated log template library includes: Performing word segmentation on the unmatched log data to obtain word frequency information corresponding to the unmatched log data; According to the word frequency information corresponding to the unmatched log data, the preset word frequency library is updated to obtain an updated word frequency library; According to the word frequency information corresponding to the unmatched log data, the unmatched log data is sorted according to the updated word frequency library to obtain a sequence corresponding to the unmatched log data; The sequence corresponding to the unmatched log data is merged with the multiple template sequences corresponding to the preset log template library to obtain the updated log template library.

6. The method according to any one of claims 4-5, characterized in that The method further comprises: If the number of the multiple template sequences corresponding to the log template library is greater than a preset number, the support of each template sequence is determined according to the number of log hits and the log hit time of each template sequence, thereby obtaining multiple support degrees; the log template library is the preset log template library or the updated log template library; According to the multiple support degrees, multiple template sequences corresponding to the log template library are screened to obtain a maintained log template library, and the maintained log template library is used for a matching process of a next log data set.

7. The method according to claim 6, characterized in that The step of screening the multiple template sequences corresponding to the log template library according to the multiple support degrees to obtain a maintained log template library includes: Taking the template sequences with support greater than a preset support among the multiple template sequences as multiple candidate template sequences; If the number of the plurality of candidate template sequences is less than or equal to the preset number, determining the maintained log template library according to the plurality of candidate template sequences; If the number of the plurality of candidate template sequences is greater than the preset number, calculating vector similarities between template sequences having support less than or equal to the preset support among the plurality of support sequences and the plurality of candidate template sequences to obtain a plurality of vector similarities; Screening out a plurality of target template sequences from the plurality of candidate template sequences according to the plurality of vector similarities; wherein the number of the plurality of target template sequences is less than or equal to the preset number; The maintained log template library is determined according to the multiple target template sequences.

8. The method according to any one of claims 1 to 5, characterized in that The obtaining of the log data set includes: Get the original log dataset; Regularizing the original log data set to obtain a target log data set; Deduplication is performed on the target log data set to obtain the log data set.

9. A data processing device, characterized in that: The device comprises: Acquisition module, used to obtain log data sets; A matching module, configured to match the log data set with a preset log template library to obtain a matching degree; A clustering module, configured to determine a log clustering result based on the matching degree; A prediction module is used to determine, after receiving the alarm information, the fault prediction information corresponding to the log clustering result based on the log clustering result and a preset multi-label classification model; The preset multi-label classification model is a single-layer model; The prediction module is also used to embed the log clustering results to obtain features to be classified; Input the features to be classified into a preset multi-label classification model to obtain potential problem labels and problem types; the fault prediction information is a multi-level prediction result, which includes potential problem labels and problem types; The acquisition module is further used to obtain the true value of the fault information corresponding to the log clustering result; The training module is used to update the preset multi-label classification model based on the true value of the fault information and the fault prediction information using a preset loss function to obtain an updated multi-label classification model. The updated multi-label classification model is used to determine the fault prediction information of the next log clustering result; The fault information truth value includes the potential problem label truth value and the problem type truth value; The training module is further configured to embed and encode the true value of the problem type and the true value of the potential problem label, respectively, to obtain a type vector and an initial label vector; perform dimension conversion on the initial label vector to obtain a label vector, so that the dimension of the label vector is consistent with that of the type vector; and update the preset multi-label classification model using a preset loss function based on the type vector and label vector corresponding to the log clustering result and the fault prediction information to obtain an updated multi-label classification model; The preset loss function is the product of the loss function corresponding to the potential problem label and the first preset coefficient, and the sum of the product of the loss function corresponding to the problem type and the second preset coefficient.

10. A data processing device, characterized in that: The device comprises: a memory for storing executable computer programs; A processor, configured to implement the method according to any one of claims 1 to 8 when executing the executable computer program stored in the memory.

11. A computer-readable storage medium, characterized in that A computer program is stored, which is used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images