Log anomaly detection method based on hybrid expert network

Abstract

The invention discloses a log anomaly detection method based on a hybrid expert network, and the method comprises the following steps: S1, collecting and preprocessing log data of a plurality of heterogeneous systems, and generating unified structured feature representation; s2, inputting the data into a hybrid expert network, analyzing different log features and performing preliminary anomaly recognition; s3, dynamically adjusting expert module parameters through a self-organizing feature competition mechanism; s4, mapping the log features to a non-Euclidean space by adopting hybrid topological mapping, and processing anomalies in the high-dimensional data; s5, fusing a plurality of detection results through a recursive resonance fusion algorithm, and extracting a global abnormal mode; s6, based on a detection result, adjusting an expert network structure and parameters by using an adaptive feature feedback mechanism, and optimizing a module contribution degree; and S7, dynamically adjusting resource allocation according to the system load. The log abnormity in the complex IT system can be efficiently detected, and the detection precision and adaptability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid expert networks, and in particular to a log anomaly detection method based on a hybrid expert network. Background Art

[0002] In today's complex IT systems and big data environments, log data is an important basis for operation and maintenance management and security monitoring. Whether it is a cloud computing platform, large-scale enterprise IT infrastructure, or IoT devices, these systems generate a large amount of log data every day, recording key information such as system operation status, user operations, and fault information. Analyzing and detecting anomalies in these log data can effectively help operation and maintenance personnel and security teams to promptly discover potential system problems, abnormal behaviors, and security threats.

[0003] In the prior art, log anomaly detection methods are mainly divided into two categories: rule-based methods and machine learning-based methods. The rule-based method is the most traditional and widely used method, which detects anomalies by manually defining a series of detection rules. Although this method can show high detection accuracy when facing fixed-pattern anomalies, its main drawback is the lack of dynamic adaptability. The structure of log data is complex and changes rapidly over time. It is difficult for rules to cover all potential anomalies, especially when facing unknown attack modes or new types of faults. Traditional rule-based methods often miss key anomaly information or generate a large number of false alarms. In addition, rule-based methods also require a lot of human resources to maintain and update the rules, which makes it difficult to cope with the challenges brought by the growth of log data in modern systems.

[0004] Log anomaly detection methods based on machine learning have received widespread attention in recent years. This type of method models historical log data and uses supervised learning or unsupervised learning methods to discover abnormal patterns. Supervised learning methods require a large amount of labeled log data for training, and the detection effect of the model is heavily dependent on the quality and coverage of the training data. However, in practical applications, labeling a large amount of log data is very time-consuming and difficult, especially when faced with high-dimensional, multi-source heterogeneous data, the labeling work is almost impossible to complete. In addition, machine learning models often show poor generalization ability when faced with rapid changes in data patterns, and it is difficult to effectively deal with new anomalies caused by dynamic changes in the system.

[0005] In unsupervised learning, anomaly detection models do not rely on labeled data, but instead detect anomalies through the intrinsic distribution characteristics of the data. Although this type of method avoids the reliance on labeled data, due to the diversity and complexity of log data, unsupervised methods often find it difficult to construct sufficiently accurate feature representations when processing high-dimensional data. Traditional unsupervised methods such as clustering algorithms and dimensionality reduction algorithms can detect anomalies to a certain extent, but when processing the multi-dimensional features of log data, they have the defects of high model complexity and poor real-time performance, and they can often only detect partial abnormal patterns and cannot form a global abnormal analysis of the entire system.

[0006] In order to meet these challenges, log anomaly detection methods combined with deep learning technology have emerged in recent years. Deep learning methods reduce the workload of manual feature engineering by automatically extracting features and improve the ability to model complex log data. However, deep learning methods still have some limitations: deep learning models require a large amount of training data, which is difficult to obtain in practical applications; the model has high computational complexity and long training time, making it difficult to process massive log data generated at a high frequency in real time; deep learning models have weak detection capabilities for new types of anomalies, and once the model is trained, it is difficult to make adaptive adjustments without retraining. Therefore, how to provide a log anomaly detection method based on a hybrid expert network is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] One purpose of the present invention is to propose a log anomaly detection method based on a hybrid expert network. The present invention makes full use of the hybrid expert network, multi-dimensional feature processing, self-organizing feature competition mechanism and recursive resonance fusion algorithm, and describes in detail how to realize intelligent anomaly detection in complex log data, and has the advantages of strong dynamic adaptability, high detection accuracy, and the ability to process high-dimensional sparse data.

[0008] The log anomaly detection method based on a hybrid expert network according to an embodiment of the present invention comprises the following steps:

[0009] S1. Collect and preprocess log data from multiple heterogeneous systems. The log data is cleaned, normalized, and feature extracted to generate a unified structured feature representation to meet the needs of multi-source data input;

[0010] S2. Input the preprocessed log data into a hybrid expert network, wherein the network includes multiple expert modules, each of which analyzes different log features and performs preliminary anomaly identification;

[0011] S3, through the self-organizing feature competition mechanism, adjust the parameters of each expert module according to the feature complexity of the log data;

[0012] S4, based on the high-dimensional features of log data, a hybrid topological mapping mechanism is used to map multi-dimensional log features to non-Euclidean space to handle abnormal patterns in high-dimensional and sparse data;

[0013] S5. After each expert module completes the preliminary detection, a recursive resonance fusion algorithm is used to fuse multiple detection results to extract global abnormal patterns and avoid local misjudgment;

[0014] S6. Based on the detection results, the adaptive feature feedback mechanism is used to adjust the structure and parameters of the expert network and optimize the module contribution to cope with new abnormal patterns;

[0015] S7. According to the system load and the amount of log data, adjust the resource allocation of the expert network to balance the allocation of computing resources and detection tasks.

[0016] Optionally, S2 includes the following steps:

[0017] S21, the preprocessed log data is processed according to the feature vector X = [x1, x2, ..., x n ] to perform feature decomposition, wherein the feature vector includes time features, frequency features, spatial topology features and behavior pattern features, and generates a specific subset feature X k =[x k1 ,x k2 ,...,x km ], where k represents the log feature dimension, x kj represents the jth feature of a specific dimension;

[0018] S22, the feature vector X k Input to different expert modules, including:

[0019] S221, expert module based on time series analysis, using long short-term memory network to process time series features;

[0020] S222, expert module based on spatial topological features, using graph neural network to analyze the structural relationship between logs:

[0021] S223, expert module based on frequency domain analysis, using fast Fourier transform to analyze frequency domain features;

[0022] S224, expert module based on behavioral pattern recognition, using variational autoencoder (VAE) to identify complex behavioral patterns;

[0023] S23, each expert module receives a specific feature subset X k, and perform feature learning and anomaly detection according to the model structure. The time series module captures anomalies in time features through LSTM, the spatial topology module identifies abnormal relationships in the structure through GNN, the frequency module analyzes periodic anomalies in frequency through FFT, and the behavior pattern module identifies abnormal changes in behavior patterns through VAE;

[0024] S24, the preliminary detection results generated by each expert module are expressed as Y k =P(A k |X k ), where A k Indicates that in the feature subset X k The abnormal events detected on k is the probability of the corresponding anomaly;

[0025] S25, preliminary test results Y of each expert module k Input recursive resonance fusion algorithm, perform multi-level fusion, based on weight adjustment mechanism W k Update the contribution of each module:

[0026]

[0027] Among them, w k is the weight of the kth expert module, β k is a specific parameter related to the module, M is the total number of modules, ∑w k =1;

[0028] S26, generating a global anomaly pattern through the results of recursive resonance fusion, combining the detection results and weights of each expert module, and accurately fusing the results according to the anomaly features of different log dimensions;

[0029] S27. The weight distribution of the expert module is dynamically adjusted to optimize resource allocation based on feature complexity, detection accuracy, and system load, maintain the adaptability to multi-source log data, and provide feedback to the adaptive feature feedback mechanism.

[0030] Optionally, S3 includes the following steps:

[0031] S31, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into multiple feature subsets X k =x k1 ,x k2 ,...,x km ], each feature subset corresponds to different feature dimensions such as time, space, frequency, and behavior pattern. The complexity of the feature subset C(X k ) is calculated based on the multidimensional variation of the features:

[0032]

[0033] Among them, m is the dimension number of feature subset, Represents the second-order rate of change of the feature in the time dimension, represents the change in spatial dimension, γ and δ are the balance coefficients, is the time average characteristic value, T is the time window length of the characteristic;

[0034] S32, according to the characteristic complexity C(X k ) Start the self-organizing feature competition mechanism. Under the self-organizing feature competition mechanism, each expert module E in the hybrid expert network k Task competition is performed based on feature complexity, and each module is evaluated by a performance function π k To assess detection capabilities:

[0035]

[0036] in, Represents the detection error, T k is the response time of the detection, α1 and α2 are the adjustment weights;

[0037] S33, parameter θ of the expert module k The following formula is used to adjust and optimize the detection capability of specific features:

[0038]

[0039] Among them, η is the learning rate, β is the influence coefficient of feature complexity on parameter adjustment;

[0040] S34, according to the characteristic complexity C(X k ) and the performance function π of the expert module k , dynamically adjust the weight w of each expert module k :

[0041]

[0042] Among them, α3 is the weight adjustment coefficient, M is the total number of expert modules, Indicates the comprehensive value of complexity and performance of all modules;

[0043] S35, according to the weight w of the expert module k Allocate computing resources. The allocation formula is:

[0044]

[0045] Among them, R k represents the amount of resources allocated to the kth expert module, R tot is the total resource volume, wk is the weight of the current module, π k is the performance function;

[0046] S36. After the detection task is completed, the parameters and weights of each expert module are further optimized through the feedback mechanism, combined with the detection error and characteristic complexity C(X k ), adjust the contribution of each module in real time;

[0047] S37. The self-organizing feature competition mechanism optimizes the weights and parameters of each expert module through multiple rounds of iterations.

[0048] Optionally, S4 includes the following steps:

[0049] S41, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into several high-dimensional feature subsets X k =[x k1 ,x k2 ,...,x km ], each feature subset X k Representing features of different dimensions, multi-scale feature mapping methods are applied to different feature subsets, combining local and global information for processing;

[0050] S42. For each feature subset X k , the features are transformed from Euclidean space to non-Euclidean space through a non-Euclidean mapping function:

[0051]

[0052] Among them, T k is the non-Euclidean feature after mapping, M k is the feature mapping matrix, B k is the offset vector, γ is the global adjustment factor, d ij represents the distance between feature points i and j, λ k is the Lagrange multiplier characterized by ;

[0053] S43. After mapping to non-Euclidean space, the topological structure of the feature subset is optimized using the Laplace Beltrami operator:

[0054]

[0055] Where Δf(x) represents the Laplace Beltrami change of feature point x, ρ is the control coefficient, ζ is the regularization coefficient, and T k ' is the optimized topological feature;

[0056] S44, for the mapped and optimized feature subset Tk Perform aggregation processing, combine the multi-layer features of high-dimensional data, and fuse the mapping results of all subsets:

[0057]

[0058] Among them, α k and β k is the weight coefficient of the subset feature, g(T k ) is the feature change rate after mapping for each subset;

[0059] S45, in the fused feature space, detecting abnormal patterns by analyzing the dynamic changes of each feature subset;

[0060] S46, combining the feature change rate and the local disturbance, feeding back the detected abnormal pattern to the adaptive feature feedback mechanism to further optimize the parameters and detection strategy used for topological mapping in the expert network;

[0061] S47, through multiple rounds of mapping and optimization iterations, dynamically adjust the weight coefficient α of each subset according to the changes of different feature subsets k and β k .

[0062] Optionally, S5 includes the following steps:

[0063] S51, after completing the preliminary detection, each expert module generates a detection result Y k =P(A k |X k ), where Y k For the kth module based on feature subset X k The anomaly detection result of A k Indicates a specific abnormal event;

[0064] S52. Multiple detection results are fused through a recursive resonance fusion algorithm. The recursive resonance mechanism generates a global abnormal pattern by measuring the interaction between the detection results of different modules:

[0065]

[0066] Among them, α k is the module weight, η is the resonance enhancement coefficient, cosine(Y i ,Y j ) represents the cosine similarity of the detection results of two modules;

[0067] S53, the recursive resonance fusion mechanism introduces the second-order derivative characteristics in the time dimension to capture the changes in abnormal patterns over time:

[0068]

[0069] Among them, F t is the global anomaly pattern after fusion in the time dimension, γ t is the time recursion coefficient, δ is the time smoothing coefficient, Represents the second-order rate of change of the abnormal pattern in the time dimension;

[0070] S54, the fused global anomaly pattern is generated through multi-level resonance characteristics, combining local anomaly detection results and time-recursive resonance patterns:

[0071]

[0072] Among them, λ k is the contribution of each expert module, ν is the resonance correction coefficient;

[0073] S55. In the recursive resonance fusion process, the global abnormal mode A of each iteration g By recursively adjusting the resonance enhancement coefficient η and the time recursion coefficient γ t :

[0074]

[0075] Among them, α is the adjustment step size, which is used to adaptively adjust the strength of the recursive resonance;

[0076] S56. Through the adaptive weight adjustment mechanism, adjust the contribution λ of each module according to the detection performance and resonance degree of each module k and the global resonance coefficient ν;

[0077] S57, after multiple rounds of recursive iterations, the recursive resonance fusion algorithm combines the detection results of all modules to generate the final global anomaly pattern A g , and feeds this pattern back to the system adaptive feature feedback mechanism to further optimize the detection parameters and strategies of the expert module.

[0078] Optionally, the S6 comprises the following steps:

[0079] S61. Global abnormal pattern A generated based on recursive resonance fusion algorithm g , analyze the contribution of each expert module in global detection k ,Adjust the feature feedback weight of each module according to the difference in detection performance;

[0080] S62, the adaptive feature feedback mechanism adjusts the weight λ of each expert module k , optimize the response to abnormal patterns, combined with the feature complexity C(X k ) and detection performance π k Dynamically update weights;

[0081] S63, through the feedback mechanism, the model parameters θ of the expert module are continuously adjusted according to the detection results k , adaptive optimization based on the complexity of specific features;

[0082] S64, the system introduces a feature noise adjustment module through an adaptive feature feedback mechanism. When processing sparse features and abnormal patterns, the feature noise level N (X k ) Adaptively adjust the module contribution, the adjustment formula is:

[0083] S65, adaptive feature feedback mechanism based on feature change rate Adjust the response speed of the expert module to quickly respond to new abnormal patterns;

[0084] S66, through iterative feedback optimization, continuously adjust the contribution of the expert module λ according to the new detection results and changes in abnormal patterns k and parameter θ k .

[0085] The beneficial effects of the present invention are:

[0086] The present invention introduces a hybrid expert network to perform hierarchical processing on log features of different dimensions. Each expert module processes features such as time series and spatial topology respectively. Using the self-organizing feature competition mechanism, the system dynamically adjusts module parameters according to the complexity of log data to ensure that each module can work flexibly according to actual needs. This design effectively solves the problem in the prior art that a single model is difficult to cope with multi-dimensional heterogeneous log data, and improves the flexibility and accuracy of anomaly detection.

[0087] The recursive resonance fusion algorithm is used to fuse the detection results of each expert module, capture the resonance characteristics between different modules, and generate a more global abnormal pattern. The traditional weighted average or simple voting mechanism is difficult to fully utilize the synergy between modules. The present invention realizes the extraction of abnormal patterns across time dimensions through the recursive resonance mechanism, avoids local misjudgment, and ensures the global consistency and accuracy of the detection results.

[0088] The adaptive feature feedback mechanism of the present invention dynamically optimizes the structure and parameters of the expert network by analyzing the detection results in real time. Unlike the models in the prior art that lack adaptive adjustment, this mechanism can timely adjust the weights and parameter configurations of each expert module according to the new abnormal pattern, ensuring that the system can still respond efficiently when facing complex and changing log data. This feedback mechanism significantly enhances the adaptability of the system, especially when dealing with new or unknown abnormal patterns, it can make adjustments quickly and improve the detection efficiency of the system.

[0089] The present invention also adopts a hybrid topological mapping mechanism to map log features to non-Euclidean space in order to process complex topological relationships in high-dimensional sparse data. The existing technology is inefficient in processing high-dimensional data. The present invention effectively improves the ability to capture abnormal patterns in high-dimensional data through non-Euclidean space mapping, especially when there are complex interactive relationships between log features, providing a more efficient anomaly detection solution and overcoming the bottleneck of traditional technology in high-dimensional data processing.

[0090] Through the self-organizing feature competition mechanism, the present invention can adjust the contribution of expert modules according to real-time log data, ensuring that dynamic competition between modules can optimize detection performance, solving the problem that static models in the prior art are difficult to cope with changing log environments. The dynamic adjustment mechanism between expert modules enables the system to automatically adjust parameters when facing unknown or complex abnormal patterns, ensuring that the system has a high degree of adaptability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0092] Figure 1 A flow chart of the log anomaly detection method based on hybrid expert network proposed by the present invention;

[0093] Figure 2 The flowchart of the self-organizing feature competition mechanism proposed by the present invention;

[0094] Figure 3 This is a schematic diagram of the hybrid topology mapping mechanism proposed in the present invention. DETAILED DESCRIPTION

[0095] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0096] refer to Figure 1-3 ,The log anomaly detection method based on hybrid expert network includes the following steps:

[0097] S1. Collect and preprocess log data from multiple heterogeneous systems. The log data is cleaned, normalized, and feature extracted to generate a unified structured feature representation to meet the needs of multi-source data input;

[0098] S2. Input the preprocessed log data into a hybrid expert network, wherein the network includes multiple expert modules, each of which analyzes different log features and performs preliminary anomaly identification;

[0099] S3, through the self-organizing feature competition mechanism, adjust the parameters of each expert module according to the feature complexity of the log data;

[0100] S4, based on the high-dimensional features of log data, a hybrid topological mapping mechanism is used to map multi-dimensional log features to non-Euclidean space to handle abnormal patterns in high-dimensional and sparse data;

[0101] S5. After each expert module completes the preliminary detection, a recursive resonance fusion algorithm is used to fuse multiple detection results to extract global abnormal patterns and avoid local misjudgment;

[0102] S6. Based on the detection results, the adaptive feature feedback mechanism is used to adjust the structure and parameters of the expert network and optimize the module contribution to cope with new abnormal patterns;

[0103] S7. According to the system load and the amount of log data, adjust the resource allocation of the expert network to balance the allocation of computing resources and detection tasks.

[0104] In this implementation, S2 includes the following steps:

[0105] S21, the preprocessed log data is processed according to the feature vector X = [x1, x2, ..., x n ] to perform feature decomposition, wherein the feature vector includes time features, frequency features, spatial topology features and behavior pattern features, and generates a specific subset feature X k =[x k1 ,x k2 ,...,x km ], where k represents the log feature dimension, x kj represents the jth feature of a specific dimension;

[0106] S22, the feature vector X k Input to different expert modules, including:

[0107] S221, expert module based on time series analysis, using long short-term memory network to process time series features;

[0108] S222, expert module based on spatial topological features, using graph neural network to analyze the structural relationship between logs:

[0109] S223, expert module based on frequency domain analysis, using fast Fourier transform to analyze frequency domain features;

[0110] S224, expert module based on behavioral pattern recognition, using variational autoencoder (VAE) to identify complex behavioral patterns;

[0111] S23, each expert module receives a specific feature subset Xk , and perform feature learning and anomaly detection according to the model structure. The time series module captures anomalies in time features through LSTM, the spatial topology module identifies abnormal relationships in the structure through GNN, the frequency module analyzes periodic anomalies in frequency through FFT, and the behavior pattern module identifies abnormal changes in behavior patterns through VAE;

[0112] S24, the preliminary detection results generated by each expert module are expressed as Y k =P(A k |X k ), where A k Indicates that in the feature subset X k The abnormal events detected on k is the probability of the corresponding anomaly;

[0113] S25, preliminary test results Y of each expert module k Input recursive resonance fusion algorithm, perform multi-level fusion, based on weight adjustment mechanism W k Update the contribution of each module:

[0114]

[0115] Among them, w k is the weight of the kth expert module, β k is a specific parameter related to the module, M is the total number of modules, ∑w k =1;

[0116] S26, generating a global anomaly pattern through the results of recursive resonance fusion, combining the detection results and weights of each expert module, and accurately fusing the results according to the anomaly features of different log dimensions;

[0117] S27. The weight distribution of the expert module is dynamically adjusted to optimize resource allocation based on feature complexity, detection accuracy, and system load, maintain the adaptability to multi-source log data, and provide feedback to the adaptive feature feedback mechanism.

[0118] In this implementation, S3 includes the following steps:

[0119] S31, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into multiple feature subsets X k =x k1 ,x k2 ,...,x km ], each feature subset corresponds to different feature dimensions such as time, space, frequency, and behavior pattern. The complexity of the feature subset C(X k ) is calculated based on the multidimensional variation of the features:

[0120]

[0121] Among them, m is the dimension number of feature subset, Represents the second-order rate of change of the feature in the time dimension, represents the change in spatial dimension, γ and δ are the balance coefficients, is the time average characteristic value, T is the time window length of the characteristic;

[0122] S32, according to the characteristic complexity C(X k ) Start the self-organizing feature competition mechanism. Under the self-organizing feature competition mechanism, each expert module E in the hybrid expert network k Task competition is performed based on feature complexity, and each module is evaluated by a performance function π k To assess detection capabilities:

[0123]

[0124] in, Represents the detection error, T k is the response time of the detection, α1 and α2 are the adjustment weights;

[0125] S33, parameter θ of the expert module k The following formula is used to adjust and optimize the detection capability of specific features:

[0126]

[0127] Among them, η is the learning rate, β is the influence coefficient of feature complexity on parameter adjustment;

[0128] S34, according to the characteristic complexity C(X k ) and the performance function π of the expert module k , dynamically adjust the weight w of each expert module k :

[0129]

[0130] Among them, α3 is the weight adjustment coefficient, M is the total number of expert modules, Indicates the comprehensive value of complexity and performance of all modules;

[0131] S35, according to the weight w of the expert module k Allocate computing resources. The allocation formula is:

[0132]

[0133] Among them, R k represents the amount of resources allocated to the kth expert module, Rtot is the total resource volume, w k is the weight of the current module, π k is the performance function;

[0134] S36. After the detection task is completed, the parameters and weights of each expert module are further optimized through the feedback mechanism, combined with the detection error and characteristic complexity C(X k ), adjust the contribution of each module in real time;

[0135] S37. The self-organizing feature competition mechanism optimizes the weights and parameters of each expert module through multiple rounds of iterations.

[0136] In this implementation, S4 includes the following steps:

[0137] S41, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into several high-dimensional feature subsets X k =[x k1 ,x k2 ,...,x km ], each feature subset X k Representing features of different dimensions, multi-scale feature mapping methods are applied to different feature subsets, combining local and global information for processing;

[0138] S42. For each feature subset X k , the features are transformed from Euclidean space to non-Euclidean space through a non-Euclidean mapping function:

[0139]

[0140] Among them, T k is the non-Euclidean feature after mapping, M k is the feature mapping matrix, B k is the offset vector, γ is the global adjustment factor, d ij represents the distance between feature points i and j, λ k is the Lagrange multiplier characterized by ;

[0141] S43. After mapping to non-Euclidean space, the topological structure of the feature subset is optimized using the Laplace Beltrami operator:

[0142]

[0143] Where Δf(x) represents the Laplace Beltrami change of feature point x, ρ is the control coefficient, ζ is the regularization coefficient, and T k ' is the optimized topological feature;

[0144] S44, for the mapped and optimized feature subset T k Perform aggregation processing, combine the multi-layer features of high-dimensional data, and fuse the mapping results of all subsets:

[0145]

[0146] Among them, α k and β k is the weight coefficient of the subset feature, g(T k ) is the feature change rate after mapping for each subset;

[0147] S45, in the fused feature space, detecting abnormal patterns by analyzing the dynamic changes of each feature subset;

[0148] S46, combining the feature change rate and the local disturbance, feeding back the detected abnormal pattern to the adaptive feature feedback mechanism to further optimize the parameters and detection strategy used for topological mapping in the expert network;

[0149] S47, through multiple rounds of mapping and optimization iterations, dynamically adjust the weight coefficient α of each subset according to the changes of different feature subsets k and β k .

[0150] In this implementation, S5 includes the following steps:

[0151] S51, after completing the preliminary detection, each expert module generates a detection result Y k =P(A k |X k ), where Y k For the kth module based on feature subset X k The anomaly detection result of A k Indicates a specific abnormal event;

[0152] S52. Multiple detection results are fused through a recursive resonance fusion algorithm. The recursive resonance mechanism generates a global abnormal pattern by measuring the interaction between the detection results of different modules:

[0153]

[0154] Among them, α k is the module weight, η is the resonance enhancement coefficient, cosine(Y i ,Y j ) represents the cosine similarity of the detection results of two modules;

[0155] S53, the recursive resonance fusion mechanism introduces the second-order derivative characteristics in the time dimension to capture the changes in abnormal patterns over time:

[0156]

[0157] Among them, F t is the global anomaly pattern after fusion in the time dimension, γ t is the time recursion coefficient, δ is the time smoothing coefficient, Represents the second-order rate of change of the abnormal pattern in the time dimension;

[0158] S54, the fused global anomaly pattern is generated through multi-level resonance characteristics, combining local anomaly detection results and time-recursive resonance patterns:

[0159]

[0160] Among them, λ k is the contribution of each expert module, ν is the resonance correction coefficient;

[0161] S55. In the recursive resonance fusion process, the global abnormal mode A of each iteration g By recursively adjusting the resonance enhancement coefficient η and the time recursion coefficient γ t :

[0162]

[0163] Among them, α is the adjustment step size, which is used to adaptively adjust the strength of the recursive resonance;

[0164] S56. Through the adaptive weight adjustment mechanism, adjust the contribution λ of each module according to the detection performance and resonance degree of each module k and the global resonance coefficient ν;

[0165] S57, after multiple rounds of recursive iterations, the recursive resonance fusion algorithm combines the detection results of all modules to generate the final global anomaly pattern A g , and feeds this pattern back to the system adaptive feature feedback mechanism to further optimize the detection parameters and strategies of the expert module.

[0166] In this implementation, S6 includes the following steps:

[0167] S61. Global abnormal pattern A generated based on recursive resonance fusion algorithm g , analyze the contribution of each expert module in global detection k ,Adjust the feature feedback weight of each module according to the difference in detection performance;

[0168] S62, the adaptive feature feedback mechanism adjusts the weight λ of each expert module k , optimize the response to abnormal patterns, combined with the feature complexity C(X k ) and detection performance πk Dynamically update weights;

[0169] S63, through the feedback mechanism, the model parameters θ of the expert module are continuously adjusted according to the detection results k , adaptive optimization based on the complexity of specific features;

[0170] S64, the system introduces a feature noise adjustment module through an adaptive feature feedback mechanism. When processing sparse features and abnormal patterns, the feature noise level N (X k ) Adaptively adjust the module contribution, the adjustment formula is:

[0171] S65, adaptive feature feedback mechanism based on feature change rate Adjust the response speed of the expert module to quickly respond to new abnormal patterns;

[0172] S66, through iterative feedback optimization, continuously adjust the contribution of the expert module λ according to the new detection results and changes in abnormal patterns k and parameter θ k .

[0173] Embodiment 1:

[0174] At a large Internet company, the company operates multiple distributed systems and service platforms, generating billions of log data every day. These log data involve multiple dimensions such as server performance, network status, user operations, error reports, system status, etc. As the business expands, the scale of the system continues to increase, and the magnitude and complexity of log data are also growing synchronously. Especially in the face of multi-source heterogeneous data structures, traditional log anomaly detection methods can no longer identify abnormal events in a timely and accurate manner. Especially during peak hours, due to the real-time and sudden characteristics of log data, traditional rule-based detection methods often have missed reports and false positives, which seriously affects the stability and security of the system.

[0175] In order to effectively solve the above problems, the company decided to introduce the log anomaly detection method based on hybrid expert network proposed in this invention, aiming to improve the detection accuracy and real-time performance of abnormal events. This method uses hybrid expert network, multi-dimensional feature processing, self-organizing feature competition mechanism and recursive resonance fusion algorithm to achieve rapid detection of complex and multi-dimensional log anomalies through in-depth analysis of log data.

[0176] In actual application, the company first inputs the log data generated by the distributed system into the system of the present invention. The log data is preprocessed, including denoising, normalization and feature extraction, to form a unified structured feature representation. After that, the preprocessed log data is input into the hybrid expert network, which contains multiple expert modules, which are responsible for feature analysis of different dimensions, such as time series analysis, spatial topological relationship analysis and frequency domain analysis.

[0177] In an emergency, the company's system encountered a malicious attack from outside, and at the same time, the internal network had a performance bottleneck, causing multiple servers to restart abnormally. The traditional detection method only identified network performance anomalies, while the present invention not only detected network anomalies through multi-dimensional feature analysis, but also identified the association between multiple server restarts and external attack events in the time dimension through a recursive resonance fusion algorithm.

[0178] During the detection process, each module in the hybrid expert network dynamically adjusts its parameters through a self-organizing feature competition mechanism to optimize the detection performance of each module. Within the first minute of the network anomaly, the system responds quickly and detects that the features of the abnormal log data are highly complex. The system automatically allocates more computing resources to the expert modules that process these high-complexity features through a competition mechanism. The recursive resonance fusion algorithm fuses the detection results of each module, generates a global anomaly pattern, and identifies log features related to malicious attacks.

[0179] Through real-time detection of the system, it was found that the traffic of a specific IP address increased sharply in a short period of time, accompanied by an abnormal increase in the CPU usage of multiple servers. Through recursive resonance fusion, the system recognized that these features were highly correlated and detected that this was a continuous DDOS attack. Compared with traditional detection methods, the system of the present invention completed global detection within 5 minutes after the attack occurred, which is about 50% faster than the original system, and the false alarm rate was reduced from the original 10% to 3%.

[0180] Before applying the method of the present invention, the company processed an average of about 2 billion log data per day, the anomaly recognition rate of the traditional detection system was 80%, the average false alarm rate was 10%, and about 500 log anomalies were missed every day. After the introduction of the present invention, the anomaly recognition rate was increased to 95%, the false alarm rate was reduced to 3%, and the log missed reports were reduced by nearly 80%. In an actual DDOS attack detection, the system successfully detected and identified the attack source within 5 minutes through the present invention, which was 50% shorter than the average detection time of the original system. In addition, during the detection process, the traditional system often cannot respond to the dynamic changes of data features in a timely manner due to its static model design, while the present invention effectively avoids this problem through an adaptive feature feedback mechanism, so that the system can still maintain efficient detection performance under high load conditions.

[0181] Table 1 Comparative data of log anomaly detection systems

[0182]

[0183] As can be seen from the table, the detection efficiency after applying the present invention is significantly improved, and the false alarm rate and the number of missed alarms are greatly reduced. Especially in a high-load and high-complexity log environment, the system shows high adaptability and stability. The practical application and data of this embodiment prove that the hybrid expert network, self-organizing feature competition mechanism and recursive resonance fusion algorithm of the present invention effectively solve the problems of high-dimensional and sparse data processing difficulties, insufficient real-time detection capabilities and low detection accuracy in the prior art.

[0184] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A log anomaly detection method based on a hybrid expert network, characterized in that: The steps include: S1. Collect and preprocess log data from multiple heterogeneous systems. The log data is cleaned, normalized, and feature extracted to generate a unified structured feature representation to meet the needs of multi-source data input; S2. Input the preprocessed log data into a hybrid expert network, wherein the network includes multiple expert modules, each of which analyzes different log features and performs preliminary anomaly identification; S3, through the self-organizing feature competition mechanism, adjust the parameters of each expert module according to the feature complexity of the log data; S4, based on the high-dimensional features of log data, a hybrid topological mapping mechanism is used to map multi-dimensional log features to non-Euclidean space to handle abnormal patterns in high-dimensional and sparse data; S5. After each expert module completes the preliminary detection, a recursive resonance fusion algorithm is used to fuse multiple detection results to extract global abnormal patterns and avoid local misjudgment; S6. Based on the detection results, the adaptive feature feedback mechanism is used to adjust the structure and parameters of the expert network and optimize the module contribution to cope with new abnormal patterns; S7. According to the system load and the amount of log data, adjust the resource allocation of the expert network to balance the allocation of computing resources and detection tasks.

2. The log anomaly detection method based on hybrid expert network according to claim 1 is characterized in that: The S2 comprises the following steps: S21, the preprocessed log data is processed according to the feature vector X = [x1, x2, ..., x n ] to perform feature decomposition, wherein the feature vector includes time features, frequency features, spatial topology features and behavior pattern features, and generates a specific subset feature X k =[x k1 ,x k2 ,...,x km ], where k represents the log feature dimension, x kj represents the jth feature of a specific dimension; S22, the feature vector X k Input to different expert modules, including: S221, expert module based on time series analysis, using long short-term memory network to process time series features; S222, expert module based on spatial topological features, using graph neural network to analyze the structural relationship between logs: S223, expert module based on frequency domain analysis, using fast Fourier transform to analyze frequency domain features; S224, expert module based on behavioral pattern recognition, using variational autoencoders to identify complex behavioral patterns; S23, each expert module receives a specific feature subset X k , and perform feature learning and anomaly detection according to the model structure. The time series module captures anomalies in time features through LSTM, the spatial topology module identifies abnormal relationships in the structure through GNN, the frequency module analyzes periodic anomalies in frequency through FFT, and the behavior pattern module identifies abnormal changes in behavior patterns through VAE; S24, the preliminary detection result generated by each expert module is represented as Y k =P(A k |X k ), where A k Indicates that in the feature subset X k The abnormal events detected on k is the probability of the corresponding anomaly; S25, preliminary test results Y of each expert module k Input recursive resonance fusion algorithm, perform multi-level fusion, based on weight adjustment mechanism W k Update the contribution of each module: Among them, w k is the weight of the kth expert module, β k is a specific parameter related to the module, M is the total number of modules, ∑w k =1; S26, generating a global anomaly pattern through the results of recursive resonance fusion, combining the detection results and weights of each expert module, and accurately fusing the results according to the anomaly features of different log dimensions; S27. The weight distribution of the expert module is dynamically adjusted to optimize resource allocation based on feature complexity, detection accuracy, and system load, maintain the adaptability to multi-source log data, and provide feedback to the adaptive feature feedback mechanism.

3. The log anomaly detection method based on hybrid expert network according to claim 1 is characterized in that: The S3 comprises the following steps: S31, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into multiple feature subsets X k =x k1 ,x k2 ,...,x km ], each feature subset corresponds to different feature dimensions such as time, space, frequency, and behavior pattern. The complexity of the feature subset C(X k ) is calculated based on the multidimensional variation of the features: Among them, m is the number of dimensions of the feature subset, Represents the second-order rate of change of the feature in the time dimension, represents the change in spatial dimension, γ and δ are the balance coefficients, is the time average characteristic value, T is the time window length of the characteristic; S32, according to the characteristic complexity C(X k ) Start the self-organizing feature competition mechanism. Under the self-organizing feature competition mechanism, each expert module E in the hybrid expert network k Task competition is performed based on feature complexity, and each module is evaluated by a performance function π k To assess detection capabilities: in, Represents the detection error, T k is the response time of the detection, α1 and α2 are the adjustment weights; S33, parameter θ of the expert module k The following formula is used to adjust and optimize the detection capability of specific features: Among them, η is the learning rate, β is the influence coefficient of feature complexity on parameter adjustment; S34, according to the characteristic complexity C(X k ) and the performance function π of the expert module k , dynamically adjust the weight w of each expert module k : Among them, α3 is the weight adjustment coefficient, M is the total number of expert modules, Indicates the comprehensive value of complexity and performance of all modules; S35, according to the weight w of the expert module k Allocate computing resources. The allocation formula is: Among them, R k represents the amount of resources allocated to the kth expert module, R tot is the total resource volume, w k is the weight of the current module, π k is the performance function; S36. After the detection task is completed, the parameters and weights of each expert module are further optimized through the feedback mechanism, combined with the detection error and characteristic complexity C(X k ), adjust the contribution of each module in real time; S37. The self-organizing feature competition mechanism optimizes the weights and parameters of each expert module through multiple rounds of iterations.

4. The log anomaly detection method based on hybrid expert network according to claim 1 is characterized in that: The S4 comprises the following steps: S41, the feature vector X of the log data is [x1, x2, ..., x n ] is decomposed into several high-dimensional feature subsets X k =[x k1 ,x k2 ,...,x km ], each feature subset X k Representing features of different dimensions, multi-scale feature mapping methods are applied to different feature subsets, combining local and global information for processing; S42. For each feature subset X k , the features are transformed from Euclidean space to non-Euclidean space through a non-Euclidean mapping function: Among them, T k is the non-Euclidean feature after mapping, M k is the feature mapping matrix, B k is the offset vector, γ is the global adjustment factor, d ij represents the distance between feature points i and j, λ k is the Lagrange multiplier characterized by ; S43. After mapping to non-Euclidean space, the topological structure of the feature subset is optimized using the Laplace Beltrami operator: Where Δf(x) represents the Laplace Beltrami change of feature point x, ρ is the control coefficient, ζ is the regularization coefficient, and T k ' is the optimized topological feature; S44, for the mapped and optimized feature subset T k Perform aggregation processing, combine the multi-layer features of high-dimensional data, and fuse the mapping results of all subsets: Among them, α k and β k is the weight coefficient of the subset feature, g(T k ) is the feature change rate after mapping for each subset; S45, in the fused feature space, detecting abnormal patterns by analyzing the dynamic changes of each feature subset; S46, combining the feature change rate and the local disturbance, feeding back the detected abnormal pattern to the adaptive feature feedback mechanism to further optimize the parameters and detection strategy used for topological mapping in the expert network; S47, through multiple rounds of mapping and optimization iterations, dynamically adjust the weight coefficient α of each subset according to the changes of different feature subsets k and β k .

5. The log anomaly detection method based on hybrid expert network according to claim 1 is characterized in that: The S5 comprises the following steps: S51, after completing the preliminary detection, each expert module generates a detection result Y k =P(A k |X k ), where Y k For the kth module based on feature subset X k The anomaly detection results of A k Indicates a specific abnormal event; S52. Multiple detection results are fused through a recursive resonance fusion algorithm. The recursive resonance mechanism generates a global abnormal pattern by measuring the interaction between the detection results of different modules: Among them, α k is the module weight, η is the resonance enhancement coefficient, cosine(Y i ,Y j ) represents the cosine similarity of the detection results of two modules; S53, the recursive resonance fusion mechanism introduces the second-order derivative characteristics in the time dimension to capture the changes in abnormal patterns over time: Among them, F t is the global anomaly pattern after fusion in the time dimension, γ t is the time recursion coefficient, δ is the time smoothing coefficient, Represents the second-order rate of change of the abnormal pattern in the time dimension; S54, the fused global anomaly pattern is generated through multi-level resonance characteristics, combining local anomaly detection results and time-recursive resonance patterns: Among them, λ k is the contribution of each expert module, ν is the resonance correction coefficient; S55. In the recursive resonance fusion process, the global abnormal mode A of each iteration g By recursively adjusting the resonance enhancement coefficient η and the time recursion coefficient γ t : Among them, α is the adjustment step size, which is used to adaptively adjust the strength of the recursive resonance; S56. Through the adaptive weight adjustment mechanism, adjust the contribution λ of each module according to the detection performance and resonance degree of each module k and the global resonance coefficient ν; S57, after multiple rounds of recursive iterations, the recursive resonance fusion algorithm combines the detection results of all modules to generate the final global anomaly pattern A g , and feeds this pattern back to the system adaptive feature feedback mechanism to further optimize the detection parameters and strategies of the expert module.

6. The log anomaly detection method based on hybrid expert network according to claim 1 is characterized in that: The S6 comprises the following steps: S61. Global abnormal pattern A generated based on recursive resonance fusion algorithm g , analyze the contribution of each expert module in global detection k ,Adjust the feature feedback weight of each module according to the difference in detection performance; S62, the adaptive feature feedback mechanism adjusts the weight λ of each expert module k , optimize the response to abnormal patterns, combined with the feature complexity C(X k ) and detection performance π k Dynamically update weights; S63, through the feedback mechanism, the model parameters θ of the expert module are continuously adjusted according to the detection results k , adaptive optimization based on the complexity of specific features; S64, the system introduces a feature noise adjustment module through an adaptive feature feedback mechanism. When processing sparse features and abnormal patterns, the feature noise level N (X k ) Adaptively adjust the module contribution, the adjustment formula is: S65, adaptive feature feedback mechanism based on feature change rate Adjust the response speed of the expert module to quickly respond to new abnormal patterns; S66, through iterative feedback optimization, continuously adjust the contribution of the expert module λ according to the new detection results and changes in abnormal patterns k and parameter θ k .

Images