A log intelligent analysis system

Abstract

In the log intelligent analysis system provided by the present application, the data cleaning module cleans the source log data record set generated in a set time period to obtain a log data set to be analyzed; the word segmentation module performs word segmentation on the log data set to be analyzed to obtain a number of word segments and performs part-of-speech tagging; the importance estimation module estimates the local importance of each word segment based on a local importance assessment model and estimates the global importance of each word segment based on a global importance assessment model; the feature vectorization module constructs a key-value pair based on each word segment and the part of speech, local importance and global importance corresponding to each word segment, and vectorizes the key-value pair to obtain a feature vector corresponding to each word segment; the semantic analysis model performs semantic analysis on the feature vector to generate a semantic dependency sequence and clusters it through a first feature clustering model and a second feature clustering model to generate a result of the log intelligent analysis, at least improving the accuracy of the analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a log intelligent analysis system. Background Art

[0002] With the rapid development of informatization today, the amount of log data generated by various systems has exploded. These log data contain valuable information such as system operation status, user behavior patterns, and potential security threats. However, how to efficiently extract valuable information from massive log data has become a major challenge in the current technology field.

[0003] Traditional log analysis methods mainly rely on manually set rules or simple keyword matching, which is not only inefficient but also difficult to adapt to the diversity and dynamics of log data. With the rise of machine learning technology, some log analysis methods based on statistical learning have begun to emerge. They automatically learn the characteristics of log data by building models and improve the accuracy and efficiency of log analysis to a certain extent.

[0004] However, existing log analysis methods still have many shortcomings. First, log data often contains a lot of noise and redundant information. If directly analyzed, it will seriously affect the accuracy of the analysis results. Second, most of the existing word segmentation and feature extraction methods ignore the differences in the importance of words in log data, resulting in the extracted feature vectors unable to accurately reflect the true meaning of the log data. Finally, when dealing with log data with complex semantic relationships, existing clustering analysis methods often find it difficult to obtain accurate and meaningful clustering results. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a log intelligent analysis system to at least solve or alleviate the above problems existing in the prior art.

[0006] A log intelligent analysis system, comprising:

[0007] The data cleaning module is used to clean the source log data record set generated by the target data system in a set period of time to obtain the log data set to be analyzed;

[0008] A word segmentation module is used to perform word segmentation processing on the log data set to be analyzed to obtain a number of word segments and perform part-of-speech tagging;

[0009] An importance estimation module, used to estimate the local importance of each word in the log data set to be analyzed based on a local importance evaluation model and to estimate the global importance of each word in the historical log data set generated before the set period and the log data set to be analyzed based on a global importance evaluation model;

[0010] A feature vectorization module is used to construct a key-value pair based on each participle and the part of speech, local importance and global importance corresponding to each participle, and vectorize the key-value pair to obtain a feature vector corresponding to each participle;

[0011] An intelligent analysis module is deployed with a semantic analysis model, a first feature clustering model, and a second feature clustering model. Based on the semantic analysis model, semantic analysis is performed on the feature vector to generate a semantic dependency sequence. Based on the first feature clustering model, a semantic density-based clustering analysis is performed on the semantic dependency sequence to obtain a cluster family. Based on the second feature clustering model, spectral clustering is performed on the cluster family to generate the result of log intelligent analysis.

[0012] In the present application, the noise and redundant information in the log data are effectively removed through the data cleaning module to improve the purity of the analysis data; the log data is refinedly segmented through the word segmentation module, and part-of-speech tagging is performed to lay the foundation for subsequent feature extraction; through the importance estimation module, not only the importance of each word segmentation in the local log data is considered, but also its global importance is evaluated in combination with historical log data, so as to more accurately reflect the actual value of the word segmentation in the log data; through the feature vectorization module, the word segmentation and its part of speech, local importance and global importance and other information are constructed as key-value pairs, and vectorized processing is performed to obtain a feature vector that can comprehensively reflect the word segmentation characteristics; finally, through the semantic analysis model, the first feature clustering model and the second feature clustering model in the intelligent analysis module, the feature vector is deeply semantically analyzed and clustered to generate accurate and meaningful log intelligent analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0014] Figure 1 This is a structural diagram of a log intelligent analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the solution of the present application, the following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0016] Figure 1 This is a structural diagram of a log intelligent analysis system according to an embodiment of the present application. Figure 1 As shown, it includes:

[0017] The data cleaning module is used to clean the source log data record set generated by the target data system in a set period of time to obtain the log data set to be analyzed;

[0018] A word segmentation module is used to perform word segmentation processing on the log data set to be analyzed to obtain a number of word segments and perform part-of-speech tagging;

[0019] An importance estimation module, used to estimate the local importance of each word in the log data set to be analyzed based on a local importance evaluation model and to estimate the global importance of each word in the historical log data set generated before the set period and the log data set to be analyzed based on a global importance evaluation model;

[0020] A feature vectorization module is used to construct a key-value pair based on each participle and the part of speech, local importance and global importance corresponding to each participle, and vectorize the key-value pair to obtain a feature vector corresponding to each participle;

[0021] An intelligent analysis module is deployed with a semantic analysis model, a first feature clustering model, and a second feature clustering model. Based on the semantic analysis model, semantic analysis is performed on the feature vector to generate a semantic dependency sequence. Based on the first feature clustering model, a semantic density-based clustering analysis is performed on the semantic dependency sequence to obtain a cluster family. Based on the second feature clustering model, spectral clustering is performed on the cluster family to generate the result of log intelligent analysis.

[0022] In the present application, the noise and redundant information in the log data are effectively removed through the data cleaning module to improve the purity of the analysis data; the log data is refinedly segmented through the word segmentation module, and part-of-speech tagging is performed to lay the foundation for subsequent feature extraction; through the importance estimation module, not only the importance of each word segmentation in the local log data is considered, but also its global importance is evaluated in combination with historical log data, so as to more accurately reflect the actual value of the word segmentation in the log data; through the feature vectorization module, the word segmentation and its part of speech, local importance and global importance and other information are constructed as key-value pairs, and vectorized processing is performed to obtain a feature vector that can comprehensively reflect the word segmentation characteristics; finally, through the semantic analysis model, the first feature clustering model and the second feature clustering model in the intelligent analysis module, the feature vector is deeply semantically analyzed and clustered to generate accurate and meaningful log intelligent analysis results.

[0023] Optionally, when the data cleaning module cleans the source log data record set generated by the target data system in a set time period, it performs a preliminary inspection on the source log data record set, identifies missing values, abnormal values ​​and duplicate data therein, and completes the missing values, smoothes the abnormal values, and deletes the duplicate data to obtain the log data set to be analyzed.

[0024] Optionally, when completing the missing value, the log service field of the missing value is determined, so that according to the meaning of the log service field, a matching completion strategy is called to complete the missing value.

[0025] Optionally, according to the meaning of the log business field, a matching completion strategy is called to complete the missing values. According to the meaning of the log business field, multiple interpolation data sets are generated to predict the uncertainty of the missing values ​​and calculate a comprehensive estimate of the missing values ​​to complete the missing values.

[0026] To this end, the process of the above data cleaning module has the following technical benefits:

[0027] (1) The data cleaning module first performs a preliminary check on the source log data record set. This step can quickly identify missing values, outliers, and duplicate data in the data. This refined preprocessing process ensures the data quality of subsequent analysis and avoids analytical bias or erroneous conclusions caused by data problems.

[0028] Targeted completion strategy: For the identified missing values, the data cleaning module does not simply use a unified completion method, but calls a matching completion strategy based on the meaning of the log business field to which the missing value belongs. This targeted completion method can more accurately reflect the true situation of the missing data and improve the integrity and accuracy of the data.

[0029] (2) When filling in missing values, multiple interpolation data sets are further generated based on the meaning of the log business fields to predict the uncertainty of the missing values. By calculating the comprehensive estimate of the missing values, the data cleaning module can more comprehensively consider the various possible situations of missing data, thereby obtaining more reliable filling results. This processing method not only improves the accuracy of the data, but also enhances the robustness of data analysis.

[0030] (3) Through the above-mentioned refined data cleaning process, the present invention can ensure that the data input into the subsequent word segmentation, importance estimation, feature vectorization and intelligent analysis modules are of high quality and high accuracy. This helps to improve the analysis effect of the entire log intelligent analysis system, making the final generated log intelligent analysis results more accurate and meaningful.

[0031] Optionally, when the word segmentation module performs word segmentation processing on the log data set to be analyzed to obtain a number of word segments and performs part-of-speech tagging, it performs static word segmentation processing on the log data set to be analyzed based on the constructed word segmentation dictionary to obtain a number of word segments and performs part-of-speech tagging, and performs context perception on the number of word segments to obtain context information to adjust the number of word segments and parts of speech.

[0032] Therefore, the above processing of the word segmentation module in the log intelligent analysis system has the following technical advantages:

[0033] (1) The word segmentation module first performs static word segmentation on the log data set to be analyzed based on the constructed word segmentation dictionary. Since the word segmentation dictionary is specially constructed based on the characteristics of log data and domain knowledge, it can more accurately identify and segment the words in the log data, thereby improving the accuracy of word segmentation.

[0034] (2) While segmenting words, the word segmentation module also performs part-of-speech tagging on each word. The accuracy of part-of-speech tagging is crucial for subsequent feature extraction and semantic analysis. Through accurate part-of-speech tagging, the system can more accurately understand the grammatical role and semantic meaning of each word in the log data, thereby providing more valuable information for subsequent analysis.

[0035] (3) The word segmentation module not only performs static word segmentation and part-of-speech tagging, but also performs context-awareness on the word segmentation results. By considering the contextual information of the word segmentation in the log data, the system can adjust and correct the results of the preliminary word segmentation and part-of-speech tagging. This context-aware capability makes the word segmentation results more consistent with the actual context and semantics of the log data, thereby improving the accuracy and reliability of the word segmentation.

[0036] (3) Accurate word segmentation and part-of-speech tagging are the basis for subsequent feature extraction, semantic analysis, and clustering processing. Through the refined processing of the word segmentation module, the present invention can ensure that the data input to the subsequent modules is of high quality and high accuracy. This helps to improve the analysis effect of the entire log intelligent analysis system, making the final generated log intelligent analysis results more accurate and meaningful.

[0037] Optionally, when estimating the local importance of each word segment in the log data set to be analyzed based on the local importance assessment model, the semantic distance between each word segment and the preceding and following adjacent words segment is determined to generate a semantic distance index, as well as the position of each word segment in the log data set to be analyzed to generate a semantic position index, and the semantic distance index and the semantic position index are weighted to generate the local importance.

[0038] To this end, in the log intelligent analysis system, by determining the semantic distance between segmented words and the position of the segmented words in the data set, and combining weighted processing to generate local importance, it has the following technical benefits:

[0039] (1) By calculating the semantic distance between each segmentation and its adjacent segments, the system can capture the semantic association between words. Words with similar semantics often carry similar information or intent. Therefore, segments with closer semantic distances have higher importance in the local context. This method helps the system understand the semantic content of log data more accurately.

[0040] (2) The position of a word in a log data set is also a key factor in evaluating its local importance. For example, words that appear in key sentences or structures (such as error messages, exception information, etc.) tend to have higher importance. By considering the position information of word segments, the system can better understand the context structure of log data, thereby improving the accuracy of analysis.

[0041] (3) By weighting the semantic distance index and the semantic location index, the weights can be adjusted according to the actual application scenarios and requirements to achieve a more flexible analysis strategy. For example, in some cases, semantic distance may be more important than location information; in other cases, the opposite may be true. Through weighted processing, the system can better adapt to different analysis requirements and data characteristics.

[0042] (4) Local importance assessment: By estimating the local importance of each word, the system can prioritize and analyze words that have a greater impact on the overall results, thereby optimizing the performance and efficiency of the analysis process. This strategy helps reduce unnecessary computing overhead and improve the speed and accuracy of analysis.

[0043] Optionally, all the segmented words are formed into a segmented word sequence according to the order in which they appear in the log data set to be analyzed. , for each word in the word sequence , vectorize it to get the word vector The local importance evaluation model determines the semantic distance between each word and the adjacent words before and after it through the following formula to generate a semantic distance index:

[0044] According to the formula Calculate each word The previous adjacent participle The semantic distance , and according to the formula Calculate each word The adjacent participle The semantic distance , , Indicates the preceding adjacent participle The word vector of Indicates the next adjacent participle The word vector of

[0045] According to the formula calculate Semantic distance index .

[0046] Optionally, the local importance evaluation model determines the position of each word in the log data set to be analyzed by the following formula to generate a semantic position index:

[0047] , , , , are model parameters, Expressing participle The position index in the word sequence, Represents a semantic position index.

[0048] Optionally, the local importance evaluation model performs weighted processing on the semantic distance index and the semantic position index by the following formula to generate the local importance: :

[0049] , is a weight parameter used to balance semantic distance and semantic position.

[0050] Optionally, when the importance estimation module estimates the global importance of each word in the historical log data set generated before the set time period and the log data set to be analyzed based on the global importance evaluation model, it estimates the local occurrence frequency of each word in the log data set to be analyzed, and estimates the global occurrence frequency of each word in the historical log data set and the log data set to be analyzed, and performs named entity recognition on each word to analyze the strength of its dependency relationship with the business logic, and performs a fusion calculation on the local occurrence frequency, the global occurrence frequency and the dependency strength to estimate the global importance.

[0051] Optionally, all the segmented words are formed into a segmented word sequence according to the order in which they appear in the log data set to be analyzed. , for each word in the word sequence , based on the formula , estimate the local frequency of occurrence of each word in the log data set to be analyzed , is the current time, is a participle The last time it appeared, is the decay rate parameter, yes The number of times it appears in the log data set to be analyzed, is the total number of all tokens in the log data set to be analyzed.

[0052] Optionally, based on the formula Estimate the global frequency of occurrence of each word in the historical log data set and the log data set to be analyzed , yes The number of times it appears in the historical log data set and the log data set to be analyzed, is the total number of all tokens in these two datasets, The latest log update The number of new occurrences of is the total number of new additions to all participles in the most recent update.

[0053] Optionally, based on the formula , Analyze the strength of its dependency on business logic , is a participle With The frequency associated with each business logic, the denominator is the sum of all business logic related frequencies, express With The degree of dependency of business logic concepts. is the corresponding weight parameter, The value range is from 1 to J, where J is the total number of business logics. Expressing participle With The frequency of business logic concepts associated with The value range is from 1 to J. Expressing participle The sum of the times associated with all business logic.

[0054] Optionally, based on the formula , the local occurrence frequency, the global occurrence frequency and the dependency strength are fused and calculated to estimate the global importance , , , is a weight parameter used to balance the importance of the local occurrence frequency, the global occurrence frequency and the dependency strength.

[0055] To this end, the specific algorithms in the above semantic distance index, semantic position index and local importance have the following technical benefits:

[0056] (1) By calculating the semantic distance between a segmentation word and its adjacent segments, we can capture the contextual relationship between segmentations, which helps us to more accurately understand the meaning and role of each segmentation word in a sentence or paragraph. For example, in log data, the importance of a segmentation word may be affected by the semantics of its adjacent segmentations.

[0057] (2) The introduction of semantic position index enables the model to take into account the position information of the word in the log data. Different positions may represent different importance. For example, the beginning and end of the log data may contain important summaries or error information.

[0058] (3) The calculation formula of local importance achieves a balance between the semantic distance index and the semantic position index by weighted processing. This balance allows the model to consider both the semantic context and the position information of the word segmentation when evaluating the importance of the word segmentation, thereby obtaining a more comprehensive evaluation result.

[0059] (4) Through specific parameters (such as α, β, γ1, γ2, σ1, σ2, λ), the model can be adjusted and optimized according to specific application scenarios and data characteristics. This flexibility enables the model to adapt to different log analysis needs and be iteratively improved as the data changes.

[0060] Optionally, the feature vectorization module constructs a key-value pair based on each participle and the part of speech, local importance and global importance corresponding to each participle, and vectorizes the key-value pair to obtain a feature vector corresponding to each participle. Each participle is used as a key and its corresponding part of speech, local importance and global importance are used as a value to construct a key-value pair and store it in a hash table. The feature vector corresponding to each participle is obtained by mapping the hash table to a pre-constructed feature space, and the feature space includes all constructed vectors used to describe the participles and their related attributes.

[0061] Therefore, in the log intelligent analysis system, the feature vectorization module plays a key role in converting text data into a form that can be processed by the machine learning model. This module constructs each word and its part of speech, local importance, and global importance into a key-value pair, stores them in a hash table, and then maps them to the pre-constructed feature space to obtain the feature vector of each word. This has the following technical benefits:

[0062] (1) Word segmentation and its attributes (part of speech, local importance, global importance) are constructed as key-value pairs, realizing a structured representation of text data. This representation facilitates fast access and efficient calculation in subsequent processing.

[0063] Using hash tables to store key-value pairs can achieve data retrieval with constant time complexity, greatly improving the efficiency of data access. This is especially important for processing large-scale log data.

[0064] (2) The feature vector not only contains the text information of the word segmentation itself (reflected by the word segmentation key), but also integrates multi-dimensional features such as part of speech, local importance, and global importance. This multi-dimensional feature fusion makes the feature vector more comprehensive and accurate in reflecting the actual meaning and importance of the word segmentation in the log data.

[0065] Consistency of feature space: The pre-built feature space ensures that all word segmentations and their related attributes are mapped into a unified vector space, which facilitates unified processing and analysis by subsequent machine learning models.

[0066] (3) Flexible feature combination: By adjusting the composition of the key-value pairs in the hash table, different features can be flexibly combined to generate feature vectors, thereby meeting different application scenarios and analysis requirements.

[0067] Easy to expand: As log data continues to increase and change, the feature vectorization module can easily expand new word segmentations and their attributes into the hash table and map them into the feature space to maintain the continued effectiveness and adaptability of the system.

[0068] Optionally, when performing semantic analysis on the feature vector based on the semantic analysis model to generate a semantic dependency sequence, a domain knowledge graph is obtained, and feature extraction is performed on the domain knowledge graph to obtain business logic features, and the business logic features are integrated into the feature vector to enhance the feature vector to obtain an enhanced feature vector to generate a semantic dependency sequence. The domain knowledge graph is constructed based on domain entities and their relationships identified from historical case data using a random forest method, and the historical case data records a data set of actual cases or events that have occurred in the past and are related to the current log analysis task.

[0069] To this end, in the log intelligent analysis system, a technical solution is introduced to introduce domain knowledge graphs and enhance feature vectors to generate semantic dependency sequences. This has the following technical benefits:

[0070] 1. Improve the accuracy and depth of analysis

[0071] Application of domain knowledge graph: The domain knowledge graph is constructed based on the domain entities and their relationships identified from historical case data using the random forest method. It contains business logic features that are highly relevant to the current log analysis task. Integrating these business logic features into the feature vector can significantly enhance the expressiveness and semantic richness of the feature vector.

[0072] Generation of semantic dependency sequences: Semantic analysis based on enhanced feature vectors can more accurately identify semantic dependencies in log data, thereby generating more meaningful semantic dependency sequences. This sequence not only reflects the grammatical and semantic connections between word segments, but also incorporates domain-specific business logic information.

[0073] 2. Enhance the intelligence and adaptability of the system

[0074] Improved intelligent reasoning capabilities: As a carrier of prior knowledge, the domain knowledge graph provides the system with rich domain context information. This enables the system to perform deeper intelligent reasoning when performing log analysis, thereby more accurately understanding and explaining complex phenomena and problems in log data.

[0075] Adapt to the analysis needs of different fields: By building knowledge graphs in different fields, the system can flexibly adapt to the analysis needs of different fields. This adaptability makes the system have a wider range of application scenarios and stronger practical value.

[0076] 3. Improve the efficiency and real-time performance of analysis

[0077] Accelerate the feature extraction and fusion process: Using efficient algorithms such as random forests to build domain knowledge graphs can significantly accelerate the feature extraction and fusion process. This enables the system to complete complex log analysis tasks in a shorter time and improve the efficiency and real-time performance of the analysis.

[0078] Support online update and optimization: The domain knowledge graph can be continuously updated and optimized as new case data is continuously added. This online update mechanism ensures that the system can always use the latest domain knowledge for analysis and maintain the timeliness and accuracy of the analysis results.

[0079] Optionally, when the semantic dependency sequence is subjected to a semantic density-based clustering analysis based on the first feature clustering model to obtain a cluster family, semantic units are extracted from the semantic dependency sequence, and the extracted semantic units are defined as elements to identify the dependency strength between different semantic units, and a dependency strength matrix is ​​constructed accordingly. For any matrix element in the dependency strength matrix, a cluster analysis is performed based on a set semantic density clustering neighborhood radius and clustering constraints are performed based on a set clustering silhouette coefficient to obtain the cluster family.

[0080] Therefore, in the log intelligent analysis system, the first feature clustering model is used to perform clustering analysis on the semantic dependency sequence based on semantic density, which has the following technical benefits:

[0081] (1) Through semantic unit extraction and dependency strength definition, the key information and structural features in the semantic dependency sequence can be captured more accurately, thereby improving the accuracy of clustering.

[0082] (2) The construction of the dependency strength matrix makes the clustering process more transparent and explainable. By analyzing the dependency strength matrix, we can deeply understand the relationship between different semantic units and then give a reasonable explanation for the clustering results.

[0083] (3) The settings of parameters such as the semantic density clustering neighborhood radius and clustering silhouette coefficient are flexible and can be adjusted according to different analysis requirements and data characteristics, making clustering analysis more adaptable and practical.

[0084] (4) The clustering analysis method based on semantic density can efficiently process the semantic dependency sequences in large-scale log data and quickly generate meaningful clustering families. This helps system operation and maintenance personnel to quickly locate problems, understand system status, and take corresponding measures to optimize and improve.

[0085] (5) This technical solution not only considers the direct dependencies between semantic units, but also reveals deeper semantic structures through cluster analysis. This enables the system to model complex semantic relationships more accurately and provide more valuable information support for subsequent log intelligent analysis.

[0086] Optionally, when spectral clustering is performed on the cluster family based on the second feature clustering model to generate the result of log intelligent analysis, the cluster family is converted into the input of spectral clustering, feature points are extracted from the input, and the similarity between the feature points is calculated to construct a similarity matrix, the similarity matrix is ​​mapped to a Laplace matrix, and eigendecomposition is performed on the Laplace matrix to obtain eigenvalues ​​and eigenvectors, a spectral feature space is constructed based on the eigenvector, the eigenvalues ​​are mapped to the spectral feature space to obtain a target number of clusters, and the result of log intelligent analysis is generated based on the target number of clusters.

[0087] The result of intelligent log analysis is a set of clusters, each of which contains a series of similar log entries or feature points. Mathematically, this can be seen as a process of dividing the original data into several subsets. The data points in each subset have a high degree of similarity, while the data points between different subsets are relatively independent, in order to identify anomalies in log data. For example, the distance between abnormal log data and data points of normal categories is usually far, so as to quickly locate possible problems.

[0088] Therefore, in the log intelligent analysis system, a technical solution is adopted to generate analysis results by spectral clustering of cluster families based on the second feature clustering model, which has the following technical advantages:

[0089] (1) The spectral clustering algorithm can identify sample spaces of arbitrary shapes and converge to the global optimal solution, which gives it a significant advantage in processing complex and nonlinearly distributed data.

[0090] By taking the cluster families obtained from previous clustering as input, spectral clustering can further refine the clustering results and improve the accuracy and robustness of clustering.

[0091] (2) The spectral clustering algorithm transforms the complex relationships in the original data into easy-to-understand mathematical forms by constructing similarity matrices and Laplace matrices. This makes the clustering results more interpretable and helps operation and maintenance personnel understand the key information and abnormal patterns in the log data.

[0092] (3) The spectral clustering algorithm does not need to know the specific number of clusters in advance, but automatically determines the number of clusters based on the similarity of data points. This makes it more adaptable when processing unknown or dynamically changing log data.

[0093] (4) The spectral clustering algorithm maps high-dimensional data into a low-dimensional space for processing through mathematical methods such as feature decomposition. This greatly reduces the computational complexity and improves the efficiency of log intelligent analysis.

[0094] It should be noted that the models mentioned in the embodiments of the present application are not limited to uniqueness and can be flexibly selected according to the indicator requirements of the application.

[0095] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A log intelligent analysis system, characterized in that: include: The data cleaning module is used to clean the source log data record set generated by the target data system in a set period of time to obtain the log data set to be analyzed; A word segmentation module is used to perform word segmentation processing on the log data set to be analyzed to obtain a number of word segments and perform part-of-speech tagging; An importance estimation module, used to estimate the local importance of each word in the log data set to be analyzed based on a local importance evaluation model and to estimate the global importance of each word in the historical log data set generated before the set period and the log data set to be analyzed based on a global importance evaluation model; A feature vectorization module is used to construct a key-value pair based on each participle and the part of speech, local importance and global importance corresponding to each participle, and vectorize the key-value pair to obtain a feature vector corresponding to each participle; an intelligent analysis module, on which a semantic analysis model, a first feature clustering model, and a second feature clustering model are deployed; a semantic analysis is performed on the feature vector based on the semantic analysis model to generate a semantic dependency sequence; a clustering analysis based on semantic density is performed on the semantic dependency sequence based on the first feature clustering model to obtain a cluster family; and a spectral clustering is performed on the cluster family based on the second feature clustering model to generate a result of log intelligent analysis; Wherein, when the importance estimation module estimates the global importance of each word in the historical log data set generated before the set time period and the log data set to be analyzed based on the global importance evaluation model, the local occurrence frequency of each word in the log data set to be analyzed is estimated, and the global occurrence frequency of each word in the historical log data set and the log data set to be analyzed is estimated, and each word is subjected to named entity recognition to analyze the strength of its dependency relationship with the business logic, and the local occurrence frequency, the global occurrence frequency and the dependency strength are fused and calculated to estimate the global importance; Among them, all the segmented words form a segmented sequence according to the order in the log data set to be analyzed , for each word in the word sequence , based on the formula , estimate the local frequency of occurrence of each word in the log data set to be analyzed , is the current time, is a participle The last time it appeared, is the decay rate parameter, yes The number of times it appears in the log data set to be analyzed, is the total number of all word segments in the log data set to be analyzed; Among them, based on the formula Estimate the global frequency of occurrence of each word in the historical log data set and the log data set to be analyzed , yes The number of times it appears in the historical log data set and the log data set to be analyzed, is the total number of all tokens in these two datasets, The latest log update The number of new occurrences of is the total number of new additions to all participles in the most recent update; Among them, based on the formula , Analyze the strength of its dependency on business logic , is a participle With The frequency associated with each business logic, the denominator is the sum of all business logic related frequencies, express With The degree of dependency of business logic concepts. is the corresponding weight parameter, The value range is from 1 to J, where J is the total number of business logics. Expressing participle With The frequency of business logic concepts associated with The value range is from 1 to J. Expressing participle The sum of the times associated with all business logic; Among them, based on the formula , the local occurrence frequency, the global occurrence frequency and the dependency strength are fused and calculated to estimate the global importance , , , is a weight parameter used to balance the importance of the local occurrence frequency, the global occurrence frequency and the dependency strength.

2. A log intelligent analysis system according to claim 1, characterized in that: When the data cleaning module cleans the source log data record set generated by the target data system in a set time period, it performs a preliminary check on the source log data record set, identifies missing values, abnormal values ​​and duplicate data therein, and fills in the missing values, smoothes the abnormal values, and deletes the duplicate data to obtain the log data set to be analyzed.

3. A log intelligent analysis system according to claim 2, characterized in that: When the missing value is completed, the log service field of the missing value is determined, so as to call a matching completion strategy according to the meaning of the log service field to complete the missing value.

4. A log intelligent analysis system according to claim 3, characterized in that: According to the meaning of the log business field, a matching completion strategy is called to complete the missing value. According to the meaning of the log business field, multiple interpolation data sets are generated to predict the uncertainty of the missing value and calculate the comprehensive estimate of the missing value to complete the missing value.

5. A log intelligent analysis system according to claim 4, characterized in that: When the word segmentation module performs word segmentation processing on the log data set to be analyzed to obtain a number of word segments and performs part-of-speech tagging, static word segmentation processing is performed on the log data set to be analyzed based on the constructed word segmentation dictionary to obtain a number of word segments and perform part-of-speech tagging, and context perception is performed on the number of word segments to obtain context information to adjust the number of word segments and parts of speech.

6. A log intelligent analysis system according to claim 5, characterized in that: When estimating the local importance of each word in the log data set to be analyzed based on the local importance evaluation model, the semantic distance between each word and the preceding and following adjacent words is determined to generate a semantic distance index, as well as the position of each word in the log data set to be analyzed to generate a semantic position index, and the semantic distance index and the semantic position index are weighted to generate the local importance.

7. A log intelligent analysis system according to claim 6, characterized in that: All the segmented words form a segmented sequence according to the order in which they appear in the log data set to be analyzed. , for each word in the word sequence , vectorize it to get the word vector The local importance evaluation model determines the semantic distance between each word and the adjacent words before and after it through the following formula to generate a semantic distance index: According to the formula Calculate each word The previous adjacent participle The semantic distance , and according to the formula Calculate each word The adjacent participle The semantic distance , , Indicates the preceding adjacent participle The word vector of Indicates the next adjacent participle The word vector of According to the formula calculate Semantic distance index ; The local importance evaluation model determines the position of each word in the log data set to be analyzed by the following formula to generate a semantic position index: , , , , are model parameters, Expressing participle The position index in the word sequence, Represents a semantic location index; The local importance evaluation model performs weighted processing on the semantic distance index and the semantic position index through the following formula to generate the local importance: : , is a weight parameter used to balance semantic distance and semantic position.

Images