Unified monitoring log acquisition method
By extracting word and sentence similarity from log data in network systems, performing clustering and distance adjustment, and mining strong association rules, the problem of low efficiency and poor accuracy in network system fault detection in existing technologies is solved, and efficient fault diagnosis and alarm are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-04-07
AI Technical Summary
Existing network system fault detection methods are inefficient and inaccurate, especially when dealing with complex fault correlations.
By acquiring log text data, extracting word and sentence similarity between log data, performing clustering and constructing a connection graph, mining strong association rules, adjusting distances, obtaining and correcting real-time alarm parameters, and finally issuing alarms.
It achieves highly accurate and timely alarms for network system faults, improving fault diagnosis efficiency.
Smart Images
Figure CN121807658A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computers, more particularly, to a unified monitoring log collection method. BACKGROUND
[0002] With the rapid development and wide application of Internet technology, network systems have become an important infrastructure for modern social economic activities and information exchange, and their stability and reliability are crucial. However, with the continuous progress of technology and the continuous expansion of network scale, the complexity of network systems is increasing, and the interdependence between network devices is also increasing. This high degree of dependence has expanded the scope and severity of network failures, and any failure can have a significant negative impact on economic activities and user satisfaction.
[0003] The existing detection method for network system failures mainly relies on manual analysis and simple statistical methods, but these methods have many limitations when dealing with complex fault correlations, and are low in efficiency and poor in accuracy. SUMMARY
[0004] The present application provides a unified monitoring log collection method to solve the problem of low efficiency and poor accuracy of the existing detection method for network system failures, comprising: Obtain log text data, extract the word similarity and sentence similarity between each log data according to the log text data; cluster the log data according to the word similarity and sentence similarity between each log data, and construct a log data connection graph according to the clustering result; mine strong association rules between log data when a failure occurs, and adjust the distance of the log data connection graph according to the strong association rules between log data; obtain real-time log data in the log data connection graph, and determine the alarm parameters of each log data according to the real-time log data; correct the alarm parameters of the log data according to the log data connection graph, and alarm the log data according to the corrected alarm parameters.
[0005] Further, the word similarity between each log data is extracted according to the log text data, which includes: extracting log words of the log text data, encoding each log word of the log text data to obtain a log word vector; constructing a word vector sequence according to all log word vectors of the log text data, obtaining word vectors of the remaining log text data, and calculating the average cosine similarity between any word vector in the word vector sequence and the word vectors of the remaining log text data; statistics the total average cosine similarity between all word vectors of the log text data and the word vectors of the remaining log text data to obtain the word similarity between the log text and the remaining log text.
[0006] Further, the extracting the sentence similarity between each log data according to the log text data comprises: extracting log sentences of the log text data, performing sequence-to-sequence coding on the log sentences of the log text data to obtain log sentence codes; constructing a sentence code sequence according to all the log sentence codes of the log text data, obtaining the sentence codes of the remaining log text data, establishing a semantic matching model, and determining the average sentence similarity between any sentence code in the sentence code sequence and the sentence codes of the remaining log text data according to the semantic matching model; and counting the total average sentence similarity between all the sentence codes of the log text data and the sentence codes of the remaining log text data to obtain the sentence similarity between the log text and the remaining log text.
[0007] Further, the establishing the semantic matching model comprises: obtaining a preset sentence library, determining semantic pair sample data of each sentence according to the preset sentence library, and establishing a training sample set according to the semantic pair sample data; establishing an initial semantic matching model according to the training sample set and training the initial semantic matching model to obtain a trained semantic matching model.
[0008] Further, the clustering the log data according to the word similarity and the sentence similarity between each log data, and constructing a log data connection graph according to the clustering result comprises: establishing a sample data set according to all the log text data, and randomly selecting k initial clustering centers of the sample data set; obtaining preset word weights and preset sentence weights, calculating the weighted average of the word similarity and the sentence similarity between the log text data in the sample data set and the initial clustering centers according to the preset word weights and the preset sentence weights to obtain the log similarity between the log text data and the initial clustering centers; dividing each log text data into a corresponding clustering partition according to the log similarity between the log text data in the sample data set and the initial clustering centers; calculating the average log similarity between each log text data in each clustering partition, and reselecting the clustering centers according to the average log similarity between each log text data in each clustering partition; repeating the above steps until the clustering centers no longer change or the number of iterations reaches a preset maximum number of iterations to obtain the clustering result of the log text data; and setting each log data as a node, connecting the log data nodes with a log similarity greater than a first preset threshold in the same clustering cluster according to the clustering result to obtain the log data connection graph.
[0009] Furthermore, the step of mining strong correlation rules between log data when a fault occurs, and adjusting the distance of the log data connection graph based on the strong correlation rules between log data, includes: obtaining historical fault events; mining strong correlation rules between log data in the historical fault events according to the correlation rule algorithm; determining the fault association probability between log data when a fault occurs based on the strong correlation rules between log data; determining the distance between log data nodes in the log data connection graph based on the fault association probability; and adjusting the distance of the log data connection graph based on the distance between log data nodes in the log data connection graph.
[0010] Furthermore, the step of mining strong association rules between log data in historical failure events using association rule algorithms includes: determining the log text data at the time of the failure based on historical failure events, and establishing a failure feature set based on the log text data at the time of the failure; converting the failure feature set into a transactional data format to obtain a transactional failure feature set; setting a minimum support threshold and a minimum confidence threshold, and iteratively scanning the transactional failure feature set using the Apriori algorithm to generate candidate itemsets; calculating the support of each candidate itemset, and selecting frequent itemsets that meet the minimum support threshold from the candidate itemsets based on the support, and generating association rules based on the frequent itemsets; calculating the confidence of each association rule, and selecting strong association rules that meet the minimum confidence threshold from the association rules based on the confidence.
[0011] Furthermore, the step of determining the alarm parameters for each log data based on real-time log data includes: acquiring fault keywords from the real-time log data and calculating the ratio of fault keywords to all characters in the real-time log data; acquiring the changes in the ratio of fault keywords to all characters in the real-time log data and plotting a fault ratio change curve based on the changes in the ratio of fault keywords to all characters in the real-time log data; performing curve fitting on the fault change curve to obtain a fault fitting curve; determining the time required for the fault ratio to reach a third preset threshold based on the fault fitting curve; and determining the alarm parameters based on the time required for the fault ratio to reach the third preset threshold.
[0012] Furthermore, the step of correcting the alarm parameters of the log data according to the log data connection graph, and issuing an alarm for the log data based on the corrected alarm parameters, includes: obtaining a preset radius, and counting the remaining log data nodes within the preset radius of the target log data node in the log data connection graph; determining a distance parameter based on the distance between the remaining log data nodes and the target log data node, multiplying the distance parameter by the alarm parameters of the remaining log data nodes to obtain the influence parameter of the remaining log data nodes on the target log data node, and calculating the average influence parameter of all remaining log data nodes on the target log data node; obtaining the node degree of the target log data node in the log data connection graph, and determining the correction parameter based on the node degree and the average influence parameter; obtaining the alarm parameter of the target log data node, multiplying the correction parameter by the alarm parameter to obtain the corrected alarm parameter of the target log data node; and issuing an alarm for the log data when the corrected alarm parameter is greater than a fourth preset threshold.
[0013] Furthermore, determining the correction parameter based on the node degree and average influence parameter includes: determining the correction parameter according to the correction parameter calculation formula, wherein the correction parameter calculation formula is specifically as follows:
[0014] in, To correct the parameters, This is the average influence parameter. For node degree, To preset the standard node degree, For the preset range coefficient, It is a natural exponential function. This is a standardized function.
[0015] The beneficial effects of this invention are as follows: By applying the above technical solutions, this invention collects and mines text similarity and fault correlation in network system log text data, enabling timely alerts when problems occur in the log data. It boasts high accuracy and practicality, effectively improving the efficiency of fault diagnosis based on log data. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 The diagram shows the overall flowchart of a unified monitoring log collection method proposed in an embodiment of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] This application provides a unified monitoring log collection method, such as... Figure 1 As shown, it includes: S101, Obtain log text data, and extract word similarity and sentence similarity between log data based on the log text data; In some embodiments of this application, the step of extracting word similarity between log data based on log text data includes: extracting log words from the log text data; encoding each log word in the log text data to obtain log word vectors; constructing a word vector sequence based on all log word vectors in the log text data; obtaining word vectors from the remaining log text data; calculating the average cosine similarity between any word vector in the word vector sequence and the word vectors of the remaining log text data; and calculating the total average cosine similarity between all word vectors in the log text data and the word vectors of the remaining log text data to obtain the word similarity between the log text and the remaining log texts.
[0020] In this embodiment, the word vectors of each log word in the log text data are extracted and formed into a word vector sequence. The average cosine similarity between the word vector sequence and the word vectors in the remaining log texts is calculated, thereby obtaining the word similarity between the log text and the remaining log texts.
[0021] In some embodiments of this application, the step of extracting the statement similarity between log data based on log text data includes: extracting log statements from the log text data; performing sequence encoding on the log statements in the log text data to obtain log statement codes; constructing a statement encoding sequence based on all log statement codes of the log text data; obtaining the statement codes of the remaining log text data; establishing a semantic matching model; determining the average statement similarity between any statement code in the statement encoding sequence and the statement codes of the remaining log text data based on the semantic matching model; and calculating the total average statement similarity between all statement codes of the log text data and the statement codes of the remaining log text data to obtain the statement similarity between the log text and the remaining log text.
[0022] In this embodiment, a semantic matching model is established to output the total average sentence similarity between all the sentence codes of the log text data and the sentence codes of the remaining log text data, thereby obtaining the sentence similarity between the log text and the remaining log text.
[0023] In some embodiments of this application, the step of establishing a semantic matching model includes: obtaining a preset statement library; determining semantic pair sample data for each statement based on the preset statement library; establishing a training sample set based on the semantic pair sample data; establishing an initial semantic matching model based on the training sample set and training the initial semantic matching model to obtain a trained semantic matching model.
[0024] In this embodiment, a preset statement library stores several semantic pair sample data with high statement similarity. Based on the deep learning neural network model, a training sample set is established through the semantic pair sample data to build and train a semantic matching model, thereby obtaining a trained semantic matching model.
[0025] S102, cluster the log data according to the word similarity and sentence similarity between each log data, and construct a log data connection graph based on the clustering results; In some embodiments of this application, the step of clustering log data based on word and sentence similarity and constructing a log data connection graph based on the clustering results includes: establishing a sample dataset based on all log text data; randomly selecting k initial cluster centers from the sample dataset; obtaining preset word weights and preset sentence weights; calculating the weighted average of word and sentence similarity between the log text data in the sample dataset and the initial cluster centers based on the preset word weights and preset sentence weights to obtain the log similarity between the log text data and the initial cluster centers; dividing each log text data into corresponding cluster partitions based on the log similarity between the log text data in the sample dataset and the initial cluster centers; calculating the average log similarity between each log text data in each cluster partition; reselecting cluster centers based on the average log similarity between each log text data in each cluster partition; repeating the above steps iteratively until the cluster centers no longer change or the number of iterations reaches a preset maximum number of iterations to obtain the clustering results of the log text data; setting each log data as a node; connecting log data nodes with log similarity greater than a first preset threshold within the same cluster based on the clustering results to obtain a log data connection graph.
[0026] In this embodiment, the log text data is clustered based on word similarity and sentence similarity using the k-means clustering algorithm, and a log data connection graph is established based on the clustering results.
[0027] S103, Discover the strong correlation rules between log data when a fault occurs, and adjust the distance of the log data connection graph according to the strong correlation rules between log data; In some embodiments of this application, the step of mining strong correlation rules between log data when a fault occurs and adjusting the distance of the log data connection graph based on the strong correlation rules between log data includes: obtaining historical fault events; mining strong correlation rules between log data in the historical fault events according to the association rule algorithm; determining the fault association probability between log data when a fault occurs based on the strong correlation rules between log data; determining the distance between log data nodes in the log data connection graph based on the fault association probability; and adjusting the distance of the log data connection graph based on the distance between log data nodes in the log data connection graph.
[0028] In some embodiments of this application, the step of mining strong association rules between log data in historical failure events using association rule algorithms includes: determining the log text data at the time of the failure based on historical failure events, and establishing a failure feature set based on the log text data at the time of the failure; converting the failure feature set into a transactional data format to obtain a transactional failure feature set; setting a minimum support threshold and a minimum confidence threshold, and iteratively scanning the transactional failure feature set using the Apriori algorithm to generate candidate itemsets; calculating the support of each candidate itemset, and selecting frequent itemsets that meet the minimum support threshold from the candidate itemsets based on the support, and generating association rules based on the frequent itemsets; calculating the confidence of each association rule, and selecting strong association rules that meet the minimum confidence threshold from the association rules based on the confidence.
[0029] In this embodiment, the strong correlation rules between log data when a fault occurs are mined based on the Apriori algorithm. The distance between log data nodes corresponding to the fault correlation probability between each log data is obtained through a preset fault correlation probability-distance mapping graph. The higher the fault correlation probability, the larger the corresponding distance value, thereby realizing the distance adjustment of the log data connection graph.
[0030] S104, Obtain real-time log data from the log data connection diagram, and determine the alarm parameters of each log data based on the real-time log data; In some embodiments of this application, determining the alarm parameters of each log data based on real-time log data includes: obtaining fault keywords from the real-time log data and calculating the ratio of fault keywords to all characters in the real-time log data; obtaining the change in the ratio of fault keywords to all characters in the real-time log data and plotting a fault ratio change curve based on the change in the ratio of fault keywords to all characters in the real-time log data; performing curve fitting on the fault change curve to obtain a fault fitting curve; determining the time required for the fault ratio to reach a third preset threshold based on the fault fitting curve; and determining the alarm parameters based on the time required for the fault ratio to reach the third preset threshold.
[0031] In this embodiment, the time required for the fault ratio to reach the third preset threshold is calculated by extracting fault keywords from real-time log data and plotting a fault fitting curve. The reciprocal of the time required for the fault ratio to reach the third preset threshold is then determined as the alarm parameter.
[0032] S105, correct the alarm parameters of the log data according to the log data connection diagram, and issue an alarm for the log data according to the corrected alarm parameters.
[0033] In some embodiments of this application, the step of correcting the alarm parameters of log data according to the log data connection graph and issuing an alarm for the log data based on the corrected alarm parameters includes: obtaining a preset radius; counting the remaining log data nodes within the preset radius of the target log data node in the log data connection graph; determining a distance parameter based on the distance between the remaining log data nodes and the target log data node; multiplying the distance parameter by the alarm parameters of the remaining log data nodes to obtain the influence parameter of the remaining log data nodes on the target log data node; calculating the average influence parameter of all remaining log data nodes on the target log data node; obtaining the node degree of the target log data node in the log data connection graph; determining a correction parameter based on the node degree and the average influence parameter; obtaining the alarm parameter of the target log data node; multiplying the correction parameter by the alarm parameter to obtain the corrected alarm parameter of the target log data node; and issuing an alarm for the log data when the corrected alarm parameter is greater than a fourth preset threshold.
[0034] In this embodiment, a preset radius is set based on historical experience, and the remaining log data nodes within the preset radius of the target log data node are extracted. The influence parameter is calculated by the distance parameter between the remaining log data nodes and the target log data node and the alarm parameter of the remaining log data nodes. The correction parameter is obtained based on the node degree of the target log data node in the log data connection graph and the average influence parameter of all remaining log data nodes on the target log data node. The corrected alarm parameter is obtained by multiplying the corrected parameter of the target log data node with the alarm parameter. The log data is then alarmed using the corrected alarm parameter.
[0035] In some embodiments of this application, determining the correction parameter based on the node degree and average influence parameter includes: determining the correction parameter according to a correction parameter calculation formula, wherein the correction parameter calculation formula is specifically as follows:
[0036] in, To correct the parameters, This is the average influence parameter. For node degree, To preset the standard node degree, For the preset range coefficient, It is a natural exponential function. This is a standardized function.
[0037] In this embodiment, a preset range coefficient is used. Will The range is limited to [0, 2]. The correction parameter is calculated by the correction parameter calculation formula to correct the alarm parameter, so as to achieve accurate calculation of the alarm parameter of the target log data.
[0038] By applying the above technical solutions, this invention acquires log text data, extracts word and sentence similarities between log data, clusters the log data based on these similarities, and constructs a log data connectivity graph based on the clustering results. It then mines strong correlation rules between log data when a fault occurs and adjusts the distance in the connectivity graph based on these rules. Real-time log data is acquired from the connectivity graph, and alarm parameters for each log data are determined based on this data. Finally, the alarm parameters are corrected according to the connectivity graph, and alarms are triggered based on the corrected parameters. This method boasts high accuracy and practicality, effectively improving the efficiency of fault diagnosis using log data.
[0039] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A unified monitoring log collection method, characterized in that, include: Obtain log text data, and extract word similarity and sentence similarity between different log data based on the log text data; The log data are clustered based on word similarity and sentence similarity among the log data, and a log data connection graph is constructed based on the clustering results. Discover strong correlation rules between log data when a failure occurs, and adjust the distance of the log data connection graph according to the strong correlation rules between log data; Obtain real-time log data from the log data connection diagram, and determine the alarm parameters for each log data based on the real-time log data; The alarm parameters of the log data are corrected according to the log data connection diagram, and alarms are generated based on the corrected alarm parameters.
2. The unified monitoring log collection method according to claim 1, characterized in that, The step of extracting word similarity between log data based on log text data includes: Extract log terms from the log text data, encode each log term in the log text data, and obtain a log term vector; Construct a word vector sequence based on all log word vectors in the log text data, obtain the word vectors of the remaining log text data, and calculate the average cosine similarity between any word vector in the word vector sequence and the word vectors of the remaining log text data. The word similarity between the log text and the other log texts is obtained by calculating the total average cosine similarity between all word vectors of the log text data and the word vectors of the other log text data.
3. The unified monitoring log collection method according to claim 1, characterized in that, The step of extracting statement similarity between log data based on log text data includes: Extract the log statements from the log text data, and perform sequence encoding on the log statements in the log text data to obtain the log statement encoding; Construct a statement encoding sequence based on all log statement codes in the log text data, obtain the statement codes of the remaining log text data, establish a semantic matching model, and determine the average statement similarity between any statement code in the statement encoding sequence and the statement codes of the remaining log text data based on the semantic matching model. The total average sentence similarity between all the sentence codes of the log text data and the sentence codes of the other log text data is calculated to obtain the sentence similarity between the log text and the other log texts.
4. The unified monitoring log collection method according to claim 3, characterized in that, The establishment of the semantic matching model includes: Obtain a preset statement library, determine the semantic pair sample data of each statement based on the preset statement library, and build a training sample set based on the semantic pair sample data; An initial semantic matching model is established based on the training sample set, and the initial semantic matching model is trained to obtain a trained semantic matching model.
5. The unified monitoring log collection method according to claim 1, characterized in that, The process of clustering log data based on word and sentence similarity, and constructing a log data connection graph based on the clustering results, includes: A sample dataset is built based on all log text data, and k initial cluster centers are randomly selected from the sample dataset. Obtain the preset word weights and preset sentence weights, and calculate the weighted average of the word similarity and sentence similarity between the log text data in the sample dataset and the initial cluster center based on the preset word weights and preset sentence weights, so as to obtain the log similarity between the log text data and the initial cluster center; Based on the log text data in the sample dataset and the log similarity with the initial cluster centers, each log text data is divided into the corresponding cluster partition; Calculate the average log similarity among log text data within each cluster partition, and reselect cluster centers based on the average log similarity among log text data within each cluster partition; Repeat the above steps until the cluster centers no longer change or the number of iterations reaches the preset maximum number of iterations, to obtain the clustering results of the log text data; Each log data point is set as a node. Based on the clustering results, log data nodes with log similarity greater than a first preset threshold within the same cluster are connected by edges to obtain a log data connection graph.
6. The unified monitoring log collection method according to claim 1, characterized in that, When a mining failure occurs, the strong correlation rules between log data are used to adjust the distance in the log data connection graph, including: Obtain historical failure events and mine strong correlation rules between log data in the historical failure events based on the association rule algorithm; The probability of fault association between log data is determined based on the strong association rules between log data. The distance between log data nodes in the log data connection graph is determined based on the fault association probability. The distance of the log data connection graph is then adjusted based on the distance between log data nodes in the log data connection graph.
7. The unified monitoring log collection method according to claim 6, characterized in that, The method of mining strong correlation rules between log data in historical fault events based on association rule algorithms includes: Based on historical failure events, determine the log text data at the time of the failure, and establish a failure feature set based on the log text data at the time of the failure. The fault feature set is converted into a transactional data format to obtain a transactional fault feature set; Set minimum support threshold and minimum confidence threshold, and use the Apriori algorithm to iteratively scan the transactional fault feature set to generate a candidate set; Calculate the support of each candidate itemset, and select frequent itemsets that meet the minimum support threshold from the candidate itemsets based on the support. Generate association rules based on the frequent itemsets. Calculate the confidence score of each association rule, and then filter out the strong association rules that meet the minimum confidence score threshold based on the confidence score.
8. The unified monitoring log collection method according to claim 1, characterized in that, The step of determining the alarm parameters for each log data based on real-time log data includes: Obtain fault keywords from real-time log data and calculate the ratio of fault keywords to all characters in the real-time log data; Obtain the change in the ratio of fault keywords to all characters in the real-time log data, and plot the fault ratio change curve based on the change in the ratio of fault keywords to all characters in the real-time log data. The fault variation curve is fitted to obtain the fault fitting curve. The time required for the fault ratio to reach the third preset threshold is determined based on the fault fitting curve, and the alarm parameters are determined based on the time required for the fault ratio to reach the third preset threshold.
9. The unified monitoring log collection method according to claim 1, characterized in that, The step of correcting the alarm parameters of the log data according to the log data connection diagram, and issuing alarms for the log data based on the corrected alarm parameters, includes: Obtain the preset radius and count the remaining log data nodes within the preset radius of the target log data node in the log data connection graph. The distance parameter is determined based on the distance between the remaining log data nodes and the target log data node. The distance parameter is multiplied by the alarm parameter of the remaining log data nodes to obtain the influence parameter of the remaining log data nodes on the target log data node. The average influence parameter of all remaining log data nodes on the target log data node is calculated. Obtain the node degree of the target log data node in the log data connection graph, and determine the correction parameters based on the node degree and the average impact parameter; Obtain the alarm parameters of the target log data node, multiply the correction parameters by the alarm parameters, and obtain the corrected alarm parameters of the target log data node; When the corrected alarm parameter is greater than the fourth preset threshold, an alarm is triggered on the log data.
10. The unified monitoring log collection method according to claim 9, characterized in that, The determination of correction parameters based on node degree and average influence parameter includes: The correction parameters are determined according to the correction parameter calculation formula, which is as follows: in, To correct the parameters, This is the average influence parameter. For node degree, To preset the standard node degree, For the preset range coefficient, It is a natural exponential function. This is a standardized function.