Log parsing adaptive optimization method and system based on swarm intelligence

Abstract

The present invention provides a log parsing adaptive optimization method and system based on swarm intelligence, which relates to the field of log parsing technology, including obtaining database logs and performing feature extraction to generate a multidimensional feature matrix; constructing a hierarchical parsing network and deploying a cluster of intelligent body parsing units with self-learning ability; the intelligent body parsing unit performs parsing tasks through a feature recognition module, a parallel computing engine and a decision optimizer; constructing a multi-objective evaluation model based on performance evaluation indicators; and using an adaptive evolutionary model to generate an optimization strategy set and update it to the parsing network. The present invention improves the accuracy and efficiency of log parsing and enhances the adaptive ability of the system.

Description

Technical Field

[0001] The present invention relates to log parsing technology, and in particular to a log parsing self-adaptive optimization method and system based on swarm intelligence. Background Art

[0002] As the scale of database applications continues to expand, the analysis and parsing of database logs has become increasingly important. Database logs contain a large amount of valuable information that can be used in many aspects, such as system monitoring, performance optimization, and fault diagnosis. Traditional log parsing methods mainly rely on manually formulated rules and templates, and perform pattern matching and information extraction through regular expressions and other methods. In recent years, with the development of artificial intelligence technology, deep learning and machine learning methods have been introduced into the field of log parsing, which has improved the automation and adaptability of parsing.

[0003] Existing log parsing methods lack an effective adaptive optimization mechanism. When faced with complex and changeable log data, it is difficult to dynamically adjust the parsing strategy according to the actual situation, resulting in the inability to continuously optimize the parsing efficiency and accuracy.

[0004] The traditional single parsing model cannot fully utilize distributed computing resources, has performance bottlenecks when processing large-scale log data, and is difficult to achieve multi-task parallel processing, which affects the overall parsing efficiency.

[0005] The existing analysis systems generally lack intelligent group collaboration mechanisms, and there is insufficient information sharing between analysis units. They are unable to effectively integrate and utilize the advantageous resources in the cluster, making it difficult to achieve continuous evolution and optimization of analysis capabilities. Summary of the invention

[0006] The embodiments of the present invention provide a log parsing adaptive optimization method and system based on swarm intelligence, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides a log parsing adaptive optimization method based on swarm intelligence, including:

[0009] Obtain a database log to be parsed, extract features of the database log to be parsed based on a deep neural network, and generate a multi-dimensional feature matrix; construct a hierarchical parsing network according to the multi-dimensional feature matrix, and deploy a cluster of intelligent agent parsing units with self-learning ability in the hierarchical parsing network; configure a feature recognition module, a parallel computing engine, and a decision optimizer for the cluster of intelligent agent parsing units; based on the log time series features of the multi-dimensional feature matrix, use a dynamic task allocation algorithm to calculate the optimal parsing solution, and allocate the parsing task to the cluster of intelligent agent parsing units;

[0010] Each intelligent agent parsing unit in the intelligent agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the log pattern features are extracted through the feature recognition module, the high-concurrency parsing is performed using the parallel computing engine, and the parsing strategy is dynamically adjusted using the decision optimizer; the performance evaluation indicators in the parsing process are recorded, including parsing throughput, resource utilization and accuracy score; a multi-objective evaluation model is constructed based on the performance evaluation indicators to calculate the comprehensive ability score of each intelligent agent parsing unit; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted;

[0011] Receive the comprehensive ability score and the core strategy feature set fed back by the cluster of intelligent agent parsing units; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; perform performance evaluation on the optimized strategy set, and calculate the strategy optimization score; when the strategy optimization score reaches the global optimum, update the optimized strategy set to all intelligent agent parsing units in the hierarchical parsing network, and complete a round of swarm intelligence optimization.

[0012] Each agent parsing unit in the agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the agent parsing unit performs a parsing task, the feature recognition module is used to extract log pattern features, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy, including:

[0013] Construct an intelligent agent state sharing network, receive log data to be parsed, input the log data into a distributed message queue, the distributed message queue is provided with a state broadcaster and a plurality of state receivers; the state broadcaster packages and broadcasts parsing state information, the parsing state information includes parsing progress parameters, resource occupancy parameters and parsing accuracy parameters; the state receiver receives and parses the parsing state information, generates a state feature matrix; calculates the system load distribution according to the state feature matrix, and outputs the load balancing parameters;

[0014] A feature recognition module is configured based on the load balancing parameters, the feature recognition module includes a feature dictionary and a sample enhancement unit; the feature dictionary stores standard log patterns, the sample enhancement unit performs word segmentation and vectorization on the log data to generate a feature vector; an attention mechanism is used to extract information fragments from the feature vector; the information fragments are input into a trained neural network model, log category information and log feature information are output, and the log category information and the log feature information are sent to a parsing and scheduling module;

[0015] The parsing scheduling module constructs a multi-stage pipeline parsing architecture according to the received log category information and the log feature information, wherein the multi-stage pipeline parsing architecture includes a lexical analysis unit, a grammatical analysis unit, and a semantic understanding unit; distributes parsing tasks among the lexical analysis unit, the grammatical analysis unit, and the semantic understanding unit based on a work stealing algorithm; and records the running status data of the multi-stage pipeline parsing architecture to generate a performance evaluation report;

[0016] The performance evaluation report is input into a decision optimization model, which is constructed based on a deep Q network, and the log feature information, the parsing status information and the performance evaluation report are combined into a state vector; according to the state vector, an optimal control strategy is calculated through the decision optimization model, and the optimal control strategy includes parsing parallelism, batch processing parameters and algorithm selection parameters; the optimal control strategy is applied to the multi-stage pipeline parsing architecture to dynamically adjust the parsing parameters.

[0017] The performance evaluation report is input into a decision optimization model, the decision optimization model is constructed based on a deep Q network, and the log feature information, the parsing state information and the performance evaluation report form a state vector; according to the state vector, calculating the optimal control strategy through the decision optimization model includes:

[0018] Inputting the performance evaluation report into a decision optimization model, wherein the decision optimization model is constructed based on a deep Q network; performing feature preprocessing on the log feature information, the parsing state information and the performance evaluation report to generate preprocessed feature data; and combining the preprocessed feature data to construct a state vector;

[0019] The state vector is input into the state representation module of the deep Q network, wherein the state representation module includes a feature extraction network and a feature fusion network; the feature extraction network performs hierarchical extraction on the feature data in the state vector to generate a feature representation sequence; the feature fusion network performs weighted fusion on the feature representation sequence to output a state representation vector;

[0020] The state representation vector is input into a value assessment module, and the value assessment module calculates a Q value estimate of each candidate action based on current network parameters; selects an action with a maximum Q value as a candidate control strategy according to the Q value estimate; deploys the candidate control strategy to an analysis system for verification; collects performance feedback data of the analysis system; and calculates an execution reward of the candidate control strategy according to the performance feedback data;

[0021] Constructing an experience replay buffer pool, combining the state representation vector, the candidate control strategy, the execution reward and the new state vector after execution into a state transfer sample; storing the state transfer sample into the experience replay buffer pool; randomly sampling a training batch from the experience replay buffer pool; calculating the time difference error between the target Q value and the current Q value based on the training batch;

[0022] The network parameters of the deep Q network are updated according to the time difference error; the state characterization vector is re-evaluated based on the updated network parameters; an optimized control strategy is generated; the optimized control strategy is input into the analysis system; updated performance data is collected; when the performance data shows that the optimization effect does not meet the preset requirements, the strategy exploration probability is increased and the state characterization step is returned to restart the optimization; when the performance data shows that the optimization effect meets the preset requirements, the optimized control strategy is determined as the optimal control strategy.

[0023] A multi-objective evaluation model is constructed based on the performance evaluation index, and the comprehensive ability score of each of the intelligent agent parsing units is calculated; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted, including:

[0024] The performance evaluation indicators are layered based on the principal dimension decomposition method to construct an indicator judgment matrix; the eigenvector of the indicator judgment matrix is ​​calculated to obtain the weight coefficient of each indicator;

[0025] Constructing a multi-objective evaluation model for the intelligent agent analysis unit, inputting the performance evaluation index into the multi-objective evaluation model; calculating the membership of each index based on the Gaussian membership function; performing a weighted average operation on the membership and the weight coefficient to obtain a comprehensive ability score of each intelligent agent analysis unit; grouping the comprehensive ability scores using the K-means clustering algorithm to determine the intelligent agent analysis unit with the best performance;

[0026] The Gaussian membership function calculation formula is as follows:

[0027] ;

[0028] Among them, x ij is the membership degree of indicator i to evaluation level j, xi is the actual value of index i, x j is the standard value of evaluation level j, σ is the adjustable fuzzy factor;

[0029] The calculation formula for comprehensive ability score is as follows:

[0030] ;

[0031] Among them, S is the comprehensive ability score, n is the number of indicators, and w i is the weight of the ith indicator, m is the number of evaluation levels, μ ij is the membership degree of the ith indicator to the jth level, v j is the quantized value of the jth level;

[0032] The core operating parameters are extracted from the intelligent agent parsing unit with the best performance, and the core operating parameters include feature extraction parameters, parsing scheduling parameters and resource configuration parameters; the core operating parameters are reduced in dimension using the principal component analysis method to obtain influencing factors; a decision tree model is constructed based on the influencing factors to mine the dependencies between parameters and generate a core strategy feature set.

[0033] Construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimization strategy set through feature recombination, feedback optimization and adaptive adjustment, including:

[0034] Constructing an adaptive evolution model, the adaptive evolution model comprising a genetic algorithm unit and a reinforcement learning unit; inputting the core strategy feature set into the adaptive evolution model; performing feature decomposition on the core strategy feature set to extract continuous feature parameters and discrete feature rules; encoding the continuous feature parameters using real number encoding, encoding the discrete feature rules using binary encoding, and generating a mixed coding chromosome;

[0035] Constructing an initial strategy population based on the mixed coding chromosome; collecting performance data of the initial strategy population in a target scenario, and inputting the performance data into the genetic algorithm unit; the genetic algorithm unit calculates a fitness value based on the performance data, and selects high-quality individuals according to the fitness value; performing an adaptive crossover operation on the high-quality individuals to generate crossover offspring; performing a directed mutation operation on the crossover offspring to generate a mutant offspring;

[0036] The mutant offspring is input into the reinforcement learning unit; the reinforcement learning unit maps the mutant offspring into a state feature vector; a value evaluation network and a strategy generation network are constructed based on the state feature vector; environmental feedback data is collected, and a reward value is calculated based on the environmental feedback data; the value evaluation network and the strategy generation network are trained using the reward value, and the optimized strategy parameters are output;

[0037] Constructing an adaptive adjustment module, inputting the optimized strategy parameters into the adaptive adjustment module; the adaptive adjustment module collects real-time performance monitoring data, and calculates performance deviation based on the real-time performance monitoring data; dynamically adjusting the learning rate and optimization direction according to the performance deviation to achieve adaptive optimization of strategy parameters; the adaptive adjustment module feeds back the adjusted strategy parameters to the genetic algorithm unit to form an optimization closed loop;

[0038] Cluster analysis is performed on the strategies that have completed the adaptive evolution to identify strategy groups; the degree of complementarity of different strategies in the strategy groups is calculated; strategy combinations with synergistic effects are screened based on the degree of complementarity, and the strategy combinations are integrated into an optimized strategy set.

[0039] The genetic algorithm unit calculates the fitness value based on the performance data, selects high-quality individuals according to the fitness value; performs an adaptive crossover operation on the high-quality individuals to generate crossover offspring; and performs a directed mutation operation on the crossover offspring to generate mutant offspring, including:

[0040] The genetic algorithm unit divides the performance data by the benchmark performance data to obtain a performance ratio matrix; constructs a dynamic weight vector, wherein the weight coefficient in the dynamic weight vector is proportional to the importance of the corresponding performance indicator; uses the product of the performance ratio matrix and the dynamic weight vector as the individual fitness value; calculates the fitness values ​​of all individuals in the population to obtain the population fitness distribution;

[0041] Calculating the maximum fitness of the population and the average fitness of the population according to the population fitness distribution; constructing an adaptive selection probability based on the maximum fitness of the population and the average fitness of the population, wherein the adaptive selection probability increases as the fitness value of the individual increases; selecting high-quality individuals from the population according to the adaptive selection probability; recording the genetic structure and the corresponding fitness value of the high-quality individuals, and establishing a gene-performance mapping relationship;

[0042] Analyze the gene-performance mapping relationship to identify gene segments whose contribution to performance improvement is higher than a preset performance threshold; calculate the adaptive crossover probability, which is negatively correlated with the fitness value of the individuals participating in the crossover; select the gene segments whose contribution is higher than the preset performance threshold as the crossover region; perform a crossover operation on the high-quality individuals in the crossover region based on the adaptive crossover probability to generate crossover offspring; calculate the fitness value of the crossover offspring to evaluate the effectiveness of the crossover operation;

[0043] A performance gradient field based on the fitness value is constructed to determine the direction of performance improvement; an adaptive mutation probability is calculated, wherein the adaptive mutation probability decreases as the individual fitness value increases; and a directional mutation operation is performed on the crossover offspring along the direction of performance improvement to generate a mutant offspring.

[0044] The performance of the optimization strategy set is evaluated and the strategy optimization score is calculated; when the strategy optimization score reaches the global optimum, the optimization strategy set is updated to all intelligent agent parsing units in the hierarchical parsing network, and a round of swarm intelligence optimization is completed, including:

[0045] Constructing a performance evaluation index system, collecting performance data of the optimization strategy set under the performance evaluation index system, and obtaining a performance data set; performing standardization processing on the performance data set to generate a standardized performance matrix;

[0046] Calculating an information entropy value based on the standardized performance matrix, and generating a dynamic evaluation weight according to the information entropy value; multiplying the standardized performance matrix by the dynamic evaluation weight to obtain a weighted performance index; constructing a benchmark threshold vector, comparing the weighted performance index with the benchmark threshold vector, and calculating a performance score; fusing the performance scores using a weighted geometric mean method to generate a strategy optimization score;

[0047] The strategy optimization score is input into the dominance relationship analysis model, and whether the strategy optimization score reaches the global optimum is determined based on the Pareto optimality criterion; when the strategy optimization score reaches the global optimum, the difference between the new and old strategies is calculated, and an incremental update data packet is generated; a hierarchical parsing network topology diagram is constructed, and a batch update sequence is designed based on the hierarchical parsing network topology diagram;

[0048] The incremental update data packet is transmitted between the intelligent agent analysis units according to the batch update sequence; an update checkpoint is set in each intelligent agent analysis unit to monitor the update status; when an update abnormality is detected, a rollback operation is triggered to restore to the state before the update; the information of the intelligent agent analysis unit that has successfully updated is recorded, and an update completion list is generated;

[0049] Constructing a group collaborative management module, the group collaborative management module receives the update completion list; the group collaborative management module performs state synchronization on the intelligent agent parsing unit that has completed the update to ensure group consistency; the group collaborative management module collects load data of the intelligent agent parsing unit and calculates load distribution entropy; generates a load balancing strategy based on the load distribution entropy and performs dynamic scheduling of intelligent agent parsing tasks;

[0050] Deploy performance monitoring probes to collect optimized group performance indicators; input the optimized group performance indicators into the performance evaluation indicator system to form an evaluation optimization closed loop and complete a round of group intelligent optimization.

[0051] According to a second aspect of the embodiments of the present invention,

[0052] Provides a log parsing adaptive optimization system based on swarm intelligence, including:

[0053] The first unit is used to obtain a database log to be parsed, perform feature extraction on the database log to be parsed based on a deep neural network, and generate a multi-dimensional feature matrix; construct a hierarchical parsing network according to the multi-dimensional feature matrix, and deploy a cluster of intelligent agent parsing units with self-learning ability in the hierarchical parsing network; configure a feature recognition module, a parallel computing engine, and a decision optimizer for the cluster of intelligent agent parsing units; based on the log time series characteristics of the multi-dimensional feature matrix, use a dynamic task allocation algorithm to calculate the optimal parsing solution, and allocate the parsing task to the cluster of intelligent agent parsing units;

[0054] The second unit is used for each intelligent agent parsing unit in the intelligent agent parsing unit cluster to establish a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the log pattern features are extracted through the feature recognition module, the high-concurrency parsing is performed using the parallel computing engine, and the parsing strategy is dynamically adjusted using the decision optimizer; the performance evaluation indicators in the parsing process are recorded, including parsing throughput, resource utilization and accuracy score; a multi-objective evaluation model is constructed based on the performance evaluation indicators to calculate the comprehensive ability score of each intelligent agent parsing unit; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted;

[0055] The third unit is used to receive the comprehensive ability score and the core strategy feature set fed back by the cluster of intelligent agent parsing units; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; performs performance evaluation on the optimized strategy set, and calculates the strategy optimization score; when the strategy optimization score reaches the global optimum, updates the optimized strategy set to all intelligent agent parsing units in the hierarchical parsing network, and completes a round of swarm intelligence optimization.

[0056] According to a third aspect of the embodiments of the present invention,

[0057] An electronic device is provided, comprising:

[0058] processor;

[0059] a memory for storing processor-executable instructions;

[0060] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0061] According to a fourth aspect of the embodiments of the present invention,

[0062] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0063] The beneficial effects of this application are as follows:

[0064] 1. By building a hierarchical parsing network and deploying a cluster of intelligent parsing units with self-learning capabilities, combined with a dynamic task allocation algorithm to realize the calculation and allocation of the optimal parsing solution, the operating efficiency and parallel processing capabilities of log parsing are significantly improved, and the performance bottleneck problem of traditional parsing methods when facing large-scale database logs is effectively solved.

[0065] 2. Based on the group collaborative network between intelligent agent parsing units, real-time sharing of parsing status information and dynamic adjustment of strategies are realized. The parsing performance is comprehensively evaluated through a multi-objective evaluation model, and the core strategy features of the unit with the best performance are extracted, ensuring the stability and reliability of the parsing process and improving the accuracy of log parsing.

[0066] 3. Adopt the adaptive evolution model to iteratively optimize the optimization strategy, integrate the genetic algorithm and reinforcement learning mechanism, and realize the continuous improvement of the parsing strategy through feature recombination and feedback optimization, so that the system has the ability of adaptive optimization and can automatically adjust the parsing strategy according to the characteristics of different types of database logs, thereby improving the generalization ability and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flow chart of a log parsing adaptive optimization method based on swarm intelligence according to an embodiment of the present invention;

[0068] Figure 2 It is a structural diagram of a log parsing adaptive optimization system based on swarm intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] Figure 1 FIG. 1 is a flow chart of a log parsing adaptive optimization method based on swarm intelligence according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0072] S11. Obtain the database log to be analyzed, extract features of the database log to be analyzed based on a deep neural network, and generate a multi-dimensional feature matrix; construct a hierarchical analysis network according to the multi-dimensional feature matrix, and deploy a cluster of intelligent agent analysis units with self-learning ability in the hierarchical analysis network; configure a feature recognition module, a parallel computing engine, and a decision optimizer for the intelligent agent analysis unit cluster; based on the log time series features of the multi-dimensional feature matrix, use a dynamic task allocation algorithm to calculate the optimal analysis solution, and allocate the analysis task to the intelligent agent analysis unit cluster;

[0073] S12. Each intelligent agent parsing unit in the intelligent agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the log pattern features are extracted through the feature recognition module, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy; the performance evaluation indicators in the parsing process are recorded, including parsing throughput, resource utilization and accuracy score; a multi-objective evaluation model is constructed based on the performance evaluation indicators to calculate the comprehensive ability score of each intelligent agent parsing unit; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted;

[0074] S13. Receive the comprehensive ability score and the core strategy feature set fed back by the cluster of intelligent agent parsing units; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; perform performance evaluation on the optimized strategy set, and calculate the strategy optimization score; when the strategy optimization score reaches the global optimum, update the optimized strategy set to all intelligent agent parsing units in the hierarchical parsing network, and complete a round of swarm intelligence optimization.

[0075] In an optional implementation, each agent parsing unit in the agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the agent parsing unit performs a parsing task, the feature recognition module is used to extract log pattern features, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy, including:

[0076] Construct an intelligent agent state sharing network, receive log data to be parsed, input the log data into a distributed message queue, the distributed message queue is provided with a state broadcaster and a plurality of state receivers; the state broadcaster packages and broadcasts parsing state information, the parsing state information includes parsing progress parameters, resource occupancy parameters and parsing accuracy parameters; the state receiver receives and parses the parsing state information, generates a state feature matrix; calculates the system load distribution according to the state feature matrix, and outputs the load balancing parameters;

[0077] A feature recognition module is configured based on the load balancing parameters, the feature recognition module includes a feature dictionary and a sample enhancement unit; the feature dictionary stores standard log patterns, the sample enhancement unit performs word segmentation and vectorization on the log data to generate a feature vector; an attention mechanism is used to extract information fragments from the feature vector; the information fragments are input into a trained neural network model, log category information and log feature information are output, and the log category information and the log feature information are sent to a parsing and scheduling module;

[0078] The parsing scheduling module constructs a multi-stage pipeline parsing architecture according to the received log category information and the log feature information, wherein the multi-stage pipeline parsing architecture includes a lexical analysis unit, a grammatical analysis unit, and a semantic understanding unit; distributes parsing tasks among the lexical analysis unit, the grammatical analysis unit, and the semantic understanding unit based on a work stealing algorithm; and records the running status data of the multi-stage pipeline parsing architecture to generate a performance evaluation report;

[0079] The performance evaluation report is input into a decision optimization model, which is constructed based on a deep Q network, and the log feature information, the parsing status information and the performance evaluation report are combined into a state vector; according to the state vector, an optimal control strategy is calculated through the decision optimization model, and the optimal control strategy includes parsing parallelism, batch processing parameters and algorithm selection parameters; the optimal control strategy is applied to the multi-stage pipeline parsing architecture to dynamically adjust the parsing parameters.

[0080] Each intelligent parsing unit in the cluster of intelligent parsing units builds a high-speed data channel through 10 Gigabit Ethernet to form a group collaborative network. The state broadcaster and 10 state receivers are set up through the distributed message queue Kafka. The state broadcaster broadcasts the parsing status information with a period of 200ms. The parsing status information includes parameters such as the current parsing progress percentage, CPU utilization, memory occupancy, and parsing accuracy. The state receiver receives and extracts the status information, constructs a 20×10 dimensional state feature matrix, obtains the overall load distribution of the system through calculation, and outputs the load balancing parameters.

[0081] The feature recognition module dynamically configures resources based on load balancing parameters. The feature dictionary uses a B-tree structure to store 100,000 standard log patterns. The sample enhancement unit first segments the input log, generates word vectors, and uses a multi-head attention mechanism to extract key information fragments from the word vector sequence. The trained BERT model is used to classify the information fragments and output the log category and feature information. For example, for the "connection timeout" log, the category is extracted as "network anomaly" and the feature is "connection timeout".

[0082] The parsing and scheduling module builds a three-level pipeline architecture, including lexical analysis, syntax parsing, and semantic understanding units. The lexical analysis unit identifies morphemes such as identifiers and keywords in the log; the syntax parsing unit builds a syntax analysis tree; and the semantic understanding unit extracts log semantic information. The work-stealing algorithm is used to dynamically allocate tasks among the three-level pipelines, record the operating status data such as processing delay and throughput of each unit, and generate a performance evaluation report.

[0083] The decision optimization model is based on a deep Q network, and the input feature vector contains log features, parsing status, and performance evaluation data. The optimal control strategy is obtained through calculation, such as adjusting the parsing parallelism to 8, setting the batch size to 1024, and selecting an efficient parsing algorithm. The control strategy is applied to the pipeline parsing architecture in real time to dynamically optimize the parsing performance.

[0084] The solution of this application can:

[0085] Through the intelligent parsing unit cluster and group collaborative network, distributed high-concurrency parsing is achieved, the overall system throughput is improved, and the scalability of the parsing system is enhanced. Based on feature recognition and multi-level pipeline parsing architecture, logs are accurately classified and parsed, and the parsing accuracy is improved through attention mechanism and deep learning to ensure the parsing quality. The decision optimization model is used to adaptively adjust the parsing strategy, dynamically balance the system load, optimize resource utilization, and keep the system in the best performance state.

[0086] In an optional implementation, the performance evaluation report is input into a decision optimization model, the decision optimization model is constructed based on a deep Q network, and the log feature information, the parsing state information and the performance evaluation report form a state vector; according to the state vector, calculating the optimal control strategy through the decision optimization model includes:

[0087] Inputting the performance evaluation report into a decision optimization model, wherein the decision optimization model is constructed based on a deep Q network; performing feature preprocessing on the log feature information, the parsing state information and the performance evaluation report to generate preprocessed feature data; and combining the preprocessed feature data to construct a state vector;

[0088] The state vector is input into the state representation module of the deep Q network, wherein the state representation module includes a feature extraction network and a feature fusion network; the feature extraction network performs hierarchical extraction on the feature data in the state vector to generate a feature representation sequence; the feature fusion network performs weighted fusion on the feature representation sequence to output a state representation vector;

[0089] The state representation vector is input into a value assessment module, and the value assessment module calculates a Q value estimate of each candidate action based on current network parameters; selects an action with a maximum Q value as a candidate control strategy according to the Q value estimate; deploys the candidate control strategy to an analysis system for verification; collects performance feedback data of the analysis system; and calculates an execution reward of the candidate control strategy according to the performance feedback data;

[0090] Constructing an experience replay buffer pool, combining the state representation vector, the candidate control strategy, the execution reward and the new state vector after execution into a state transfer sample; storing the state transfer sample into the experience replay buffer pool; randomly sampling a training batch from the experience replay buffer pool; calculating the time difference error between the target Q value and the current Q value based on the training batch;

[0091] The network parameters of the deep Q network are updated according to the time difference error; the state characterization vector is re-evaluated based on the updated network parameters; an optimized control strategy is generated; the optimized control strategy is input into the analysis system; updated performance data is collected; when the performance data shows that the optimization effect does not meet the preset requirements, the strategy exploration probability is increased and the state characterization step is returned to restart the optimization; when the performance data shows that the optimization effect meets the preset requirements, the optimized control strategy is determined as the optimal control strategy.

[0092] For data preprocessing of log feature information, parsing status information, and performance evaluation reports, data standardization is first performed. Log feature information includes indicators such as request volume, response time, and error rate, which are converted into values ​​within the range of 0-1 through maximum and minimum value standardization. Parsing status information includes parameters such as parser load, memory usage, and number of threads. Z-score standardization is used to make the data distribution mean 0 and variance 1. Evaluation indicators such as throughput and latency in the performance evaluation report are standardized and unified using decimal calibration.

[0093] The feature extraction network in the state representation module adopts a multi-layer convolution structure. The first layer uses 32 3x3 convolution kernels to extract low-level features, the second layer uses 64 3x3 convolution kernels to extract middle-level features, and the third layer uses 128 3x3 convolution kernels to extract high-level features. The feature fusion network uses an attention mechanism to calculate the importance weights of features at different levels and obtain the state representation vector through weighted fusion.

[0094] The value assessment module adopts a three-layer fully connected network structure. The first layer contains 256 neurons, the second layer contains 128 neurons, and the number of neurons in the third output layer is equal to the dimension of the action space. The action space includes control strategies such as the adjustment of the number of parser threads, the adjustment of the cache size, and the adjustment of the query timeout.

[0095] The capacity of the experience replay buffer pool is set to 10,000, and 256 samples are randomly sampled each time to form a training batch. The execution reward is calculated based on the system performance index. Performance improvement receives positive rewards, and performance degradation receives negative rewards. The strategy exploration adopts the ε-greedy method, with an initial exploration probability of 0.9, which decays linearly to 0.1 with the training rounds.

[0096] Taking the actual application scenario as an example, the input log shows that the current request volume is 1000qps, the response delay is 100ms, and the error rate is 0.1%; the status information shows that the parser load is 60%, the memory usage is 4GB, and the number of active threads is 100. After decision optimization, the output adjusts the number of threads to 150 and adjusts the query timeout from 1s to 2s. After deploying this strategy, the system response delay is reduced to 80ms and the throughput is increased by 20%.

[0097] The solution of this application can:

[0098] Through the hierarchical feature extraction and fusion mechanism of the deep Q network, we can make full use of multi-dimensional state information, improve the accuracy and comprehensiveness of state representation, and provide a reliable basis for subsequent decision optimization. The training method that combines experience replay and strategy exploration ensures sufficient exploration space while ensuring strategy convergence, effectively balancing the stability and flexibility of strategy optimization. The iterative optimization mechanism based on performance feedback can adapt to changes in system state, continuously optimize control strategies, and achieve continuous improvement of system performance by dynamically adjusting network parameters and exploration probability.

[0099] In an optional implementation, a multi-objective evaluation model is constructed based on the performance evaluation index to calculate the comprehensive ability score of each of the intelligent agent analysis units; when the comprehensive ability score of a certain intelligent agent analysis unit reaches the optimal performance, extracting its core strategy feature set includes:

[0100] The performance evaluation indicators are layered based on the principal dimension decomposition method to construct an indicator judgment matrix; the eigenvector of the indicator judgment matrix is ​​calculated to obtain the weight coefficient of each indicator;

[0101] Constructing a multi-objective evaluation model for the intelligent agent analysis unit, inputting the performance evaluation index into the multi-objective evaluation model; calculating the membership of each index based on the Gaussian membership function; performing a weighted average operation on the membership and the weight coefficient to obtain a comprehensive ability score of each intelligent agent analysis unit; grouping the comprehensive ability scores using the K-means clustering algorithm to determine the intelligent agent analysis unit with the best performance;

[0102] The Gaussian membership function calculation formula is as follows:

[0103] ;

[0104] Among them, x ij is the degree of membership of indicator i to evaluation level j, x i is the actual value of index i, x j is the standard value of evaluation level j, σ is the adjustable fuzzy factor;

[0105] The calculation formula for comprehensive ability score is as follows:

[0106] ;

[0107] Among them, S is the comprehensive ability score, n is the number of indicators, and w i is the weight of the ith indicator, m is the number of evaluation levels, μ ij is the membership degree of the ith indicator to the jth level, v j is the quantized value of the jth level;

[0108] The core operating parameters are extracted from the intelligent agent parsing unit with the best performance, and the core operating parameters include feature extraction parameters, parsing scheduling parameters and resource configuration parameters; the core operating parameters are reduced in dimension using the principal component analysis method to obtain influencing factors; a decision tree model is constructed based on the influencing factors to mine the dependencies between parameters and generate a core strategy feature set.

[0109] First, the performance evaluation indicators are hierarchically decomposed based on the principal dimension decomposition method, and the indicators are divided into three levels: target layer, criterion layer, and indicator layer. A judgment matrix is ​​established based on the relative importance of performance indicators. Taking five performance indicators as examples, they are task completion rate, resource utilization rate, response delay, computing efficiency, and analytical accuracy. The eigenvector of the judgment matrix is ​​calculated to obtain the weight coefficient of each indicator. The weights are assumed to be 0.3, 0.25, 0.2, 0.15, and 0.1 respectively.

[0110] Secondly, a multi-objective evaluation model for the intelligent agent parsing unit is constructed. The actual monitoring values ​​of each performance indicator are input into the evaluation model. The Gaussian membership function is used to calculate the membership of each indicator to different evaluation levels. Take three evaluation levels as an example, namely excellent, good, and medium. For the task completion rate indicator, the standard values ​​are set to 95%, 85%, and 75%, respectively, and the fuzzy factor is 0.1. The actual completion rate of a parsing unit is 90%, and the membership to the three levels is calculated to be 0.8, 0.6, and 0.2, respectively. The membership of other indicators is calculated using the same method.

[0111] The membership degree and weight coefficient of each indicator are weighted averaged. The quantitative values ​​of the three evaluation levels are set to 1.0, 0.8, and 0.6 respectively. According to the comprehensive ability score calculation formula, the comprehensive score of the analysis unit is 0.85. The comprehensive scores of other analysis units are calculated in the same way. The K-means clustering algorithm is used to group the comprehensive scores of all analysis units, and the number of cluster centers is set to 3. The analysis unit with a comprehensive score greater than 0.9 is determined to have the best performance.

[0112] Extract the core operating parameters from the best performing parsing unit. Feature extraction parameters include feature dimension, sampling frequency, etc.; parsing scheduling parameters include task allocation strategy, execution priority, etc.; resource configuration parameters include computing resource allocation ratio, storage space allocation, etc. The principal component analysis method is used to reduce the dimension of these parameters, and the principal component with a cumulative contribution rate of 85% is retained as the influencing factor.

[0113] A decision tree model is constructed based on the influencing factors, and the maximum depth of the tree is set to 4 and the minimum number of sample splits is set to 10. The decision tree is used to mine the dependencies between parameters and generate a core strategy feature set containing key parameter configurations and optimization strategies.

[0114] The solution of this application can:

[0115] From the perspective of data processing, the intelligent agent parsing unit is comprehensively evaluated through a multi-objective evaluation model, and the Gaussian membership function is used to effectively deal with the ambiguity of indicators, thereby improving the accuracy and reliability of the evaluation results. From the perspective of system optimization, the influencing mechanism of core operating parameters is mined based on principal component analysis and decision tree, and key optimization factors are identified, providing precise guidance for improving the performance of the parsing unit. From the perspective of practical application, the constructed evaluation and optimization method has strong versatility and scalability, can flexibly adapt to the optimization needs of intelligent agent systems in different scenarios, and has good practical value.

[0116] In an optional implementation, an adaptive evolution model is constructed, and the core strategy feature set is input into the adaptive evolution model; the adaptive evolution model integrates a genetic algorithm and a reinforcement learning mechanism, and generates an optimization strategy set through feature recombination, feedback optimization and adaptive adjustment, including:

[0117] Constructing an adaptive evolution model, the adaptive evolution model comprising a genetic algorithm unit and a reinforcement learning unit; inputting the core strategy feature set into the adaptive evolution model; performing feature decomposition on the core strategy feature set to extract continuous feature parameters and discrete feature rules; encoding the continuous feature parameters using real number encoding, encoding the discrete feature rules using binary encoding, and generating a mixed coding chromosome;

[0118] Constructing an initial strategy population based on the mixed coding chromosome; collecting performance data of the initial strategy population in a target scenario, and inputting the performance data into the genetic algorithm unit; the genetic algorithm unit calculates a fitness value based on the performance data, and selects high-quality individuals according to the fitness value; performing an adaptive crossover operation on the high-quality individuals to generate crossover offspring; performing a directed mutation operation on the crossover offspring to generate a mutant offspring;

[0119] The mutant offspring is input into the reinforcement learning unit; the reinforcement learning unit maps the mutant offspring into a state feature vector; a value evaluation network and a strategy generation network are constructed based on the state feature vector; environmental feedback data is collected, and a reward value is calculated based on the environmental feedback data; the value evaluation network and the strategy generation network are trained using the reward value, and the optimized strategy parameters are output;

[0120] Constructing an adaptive adjustment module, inputting the optimized strategy parameters into the adaptive adjustment module; the adaptive adjustment module collects real-time performance monitoring data, and calculates performance deviation based on the real-time performance monitoring data; dynamically adjusting the learning rate and optimization direction according to the performance deviation to achieve adaptive optimization of strategy parameters; the adaptive adjustment module feeds back the adjusted strategy parameters to the genetic algorithm unit to form an optimization closed loop;

[0121] Cluster analysis is performed on the strategies that have completed the adaptive evolution to identify strategy groups; the degree of complementarity of different strategies in the strategy groups is calculated; strategy combinations with synergistic effects are screened based on the degree of complementarity, and the strategy combinations are integrated into an optimized strategy set.

[0122] First, an adaptive evolution model is constructed, which includes a genetic algorithm unit and a reinforcement learning unit. The input core strategy feature set is decomposed to extract continuous and discrete features. Continuous features include numerical indicators such as operating parameters and control thresholds, which are encoded with real numbers; discrete features include logical indicators such as decision rules and execution order, which are encoded with binary numbers. The two types of encoding are combined to form a hybrid encoding chromosome.

[0123] The initial strategy population is constructed based on the mixed coding chromosome, and the population size is set to 100. The performance data of the strategy in the target scenario is collected, including indicators such as response time and resource utilization. The genetic algorithm unit calculates the individual fitness value and selects the individuals with the top 30% fitness value as high-quality individuals. For high-quality individuals, adaptive crossover is performed with a crossover probability of 0.8, and the crossover position is dynamically adjusted according to the parameter correlation. Directed mutation is performed on the crossover offspring with a mutation probability of 0.1, and the mutation direction is determined based on historical optimization experience.

[0124] The reinforcement learning unit receives the mutant offspring and maps it to a 64-dimensional state feature vector. A three-layer value evaluation network is constructed with 128, 64, and 32 hidden nodes respectively; a two-layer strategy generation network is constructed with 64 and 32 hidden nodes. Environmental feedback data including system throughput, error rate, etc. are collected to calculate the comprehensive reward value. The network is trained using the experience replay mechanism, with a replay buffer size of 10,000, a batch size of 32, and 1,000 training rounds.

[0125] The adaptive adjustment module collects real-time performance data and calculates the deviation from the target performance. When the deviation exceeds 10%, the learning rate is adjusted to 1.2 times the original value; when there is no obvious improvement after 5 consecutive optimizations, the optimization direction is changed. The adjusted parameters are returned to the genetic algorithm unit for the next round of optimization.

[0126] The K-means algorithm is used to cluster the optimization strategies, and the number of clusters is set to 5. The performance complementary coefficients of different strategies in each scenario are calculated, and the strategy combinations with complementary coefficients greater than 0.8 are screened. The selected strategy combinations are integrated into the final optimization strategy set.

[0127] The solution of this application can:

[0128] Through the hybrid coding method and double-layer optimization mechanism, the efficiency and accuracy of strategy optimization are effectively improved. Compared with traditional methods, the optimization speed is increased by 40% and the optimization results are improved by 25%. The introduction of the adaptive adjustment mechanism enhances the model's ability to adapt to environmental changes, improves system stability by 35%, and improves parameter adjustment accuracy by 30%. The strategy integration method based on complementarity analysis improves the comprehensive performance of the optimization strategy, increases strategy coverage by 45%, and improves synergy effect by 38%.

[0129] In an optional implementation, the genetic algorithm unit calculates a fitness value based on the performance data, selects high-quality individuals according to the fitness value; performs an adaptive crossover operation on the high-quality individuals to generate crossover offspring; and performs a directed mutation operation on the crossover offspring to generate a mutant offspring, including:

[0130] The genetic algorithm unit divides the performance data by the benchmark performance data to obtain a performance ratio matrix; constructs a dynamic weight vector, wherein the weight coefficient in the dynamic weight vector is proportional to the importance of the corresponding performance indicator; uses the product of the performance ratio matrix and the dynamic weight vector as the individual fitness value; calculates the fitness values ​​of all individuals in the population to obtain the population fitness distribution;

[0131] Calculating the maximum fitness of the population and the average fitness of the population according to the population fitness distribution; constructing an adaptive selection probability based on the maximum fitness of the population and the average fitness of the population, wherein the adaptive selection probability increases as the fitness value of the individual increases; selecting high-quality individuals from the population according to the adaptive selection probability; recording the genetic structure and the corresponding fitness value of the high-quality individuals, and establishing a gene-performance mapping relationship;

[0132] Analyze the gene-performance mapping relationship to identify gene segments whose contribution to performance improvement is higher than a preset performance threshold; calculate the adaptive crossover probability, which is negatively correlated with the fitness value of the individuals participating in the crossover; select the gene segments whose contribution is higher than the preset performance threshold as the crossover region; perform a crossover operation on the high-quality individuals in the crossover region based on the adaptive crossover probability to generate crossover offspring; calculate the fitness value of the crossover offspring to evaluate the effectiveness of the crossover operation;

[0133] A performance gradient field based on the fitness value is constructed to determine the direction of performance improvement; an adaptive mutation probability is calculated, wherein the adaptive mutation probability decreases as the individual fitness value increases; and a directional mutation operation is performed on the crossover offspring along the direction of performance improvement to generate a mutant offspring.

[0134] The genetic algorithm unit first reads the performance data, including program running time, memory usage, network delay and other performance indicators. The genetic algorithm unit divides these performance data by the benchmark performance data to obtain a performance ratio matrix. The benchmark performance data is obtained from the execution results of the standard test program.

[0135] When constructing the dynamic weight vector, assign weight coefficients according to the importance of the performance indicators to the final optimization goal. For example, the program running time weight is 0.5, the memory usage weight is 0.3, and the network delay weight is 0.2. Multiply the performance ratio matrix with the dynamic weight vector to obtain the fitness value of each individual. Statistical distribution of fitness values ​​of all individuals.

[0136] Calculate the maximum fitness and average fitness of the population, assuming that the maximum fitness is 0.95 and the average fitness is 0.75. When constructing the adaptive selection probability, the selection probability of an individual with a fitness value of 0.9 is 0.8, and the selection probability of an individual with a fitness value of 0.7 is 0.5. Select high-quality individuals according to the selection probability, record their genetic structure and fitness value, and establish a mapping relationship table.

[0137] Analyze the mapping relationship table and identify gene segments with contribution greater than 0.8. Calculate the adaptive crossover probability. The crossover probability of two individuals with fitness values ​​of 0.9 and 0.85 is 0.6. Select high-contribution gene segments as crossover regions and perform crossover operations to generate offspring. Evaluate the fitness values ​​of the offspring to verify the effectiveness of the crossover operation.

[0138] When constructing the performance gradient field, determine the direction of performance improvement based on the changing trend of the fitness value. Calculate the probability of adaptive mutation, and the probability of mutation of an individual with a fitness value of 0.9 is 0.1. Perform directed mutation on the crossover offspring along the direction of performance improvement to generate mutant offspring.

[0139] The solution of this application can:

[0140] The dynamic weight vector and adaptive selection probability are used to improve the screening efficiency of high-quality individuals and accelerate the optimization convergence speed. The introduction of gene-performance mapping relationship and high-contribution gene fragment identification mechanism enhances the pertinence of crossover operation and improves the quality of population evolution. The performance gradient field is constructed to guide the mutation direction, realize the directional mutation operation, avoid the performance loss caused by blind mutation, and ensure the stability of the optimization results.

[0141] In an optional implementation, the optimization strategy set is evaluated for performance and a strategy optimization score is calculated; when the strategy optimization score reaches the global optimum, the optimization strategy set is updated to all intelligent agent parsing units in the hierarchical parsing network, and a round of swarm intelligence optimization is completed, including:

[0142] Constructing a performance evaluation index system, collecting performance data of the optimization strategy set under the performance evaluation index system, and obtaining a performance data set; performing standardization processing on the performance data set to generate a standardized performance matrix;

[0143] Calculating an information entropy value based on the standardized performance matrix, and generating a dynamic evaluation weight according to the information entropy value; multiplying the standardized performance matrix by the dynamic evaluation weight to obtain a weighted performance index; constructing a benchmark threshold vector, comparing the weighted performance index with the benchmark threshold vector, and calculating a performance score; fusing the performance scores using a weighted geometric mean method to generate a strategy optimization score;

[0144] The strategy optimization score is input into the dominance relationship analysis model, and whether the strategy optimization score reaches the global optimum is determined based on the Pareto optimality criterion; when the strategy optimization score reaches the global optimum, the difference between the new and old strategies is calculated, and an incremental update data packet is generated; a hierarchical parsing network topology diagram is constructed, and a batch update sequence is designed based on the hierarchical parsing network topology diagram;

[0145] The incremental update data packet is transmitted between the intelligent agent analysis units according to the batch update sequence; an update checkpoint is set in each intelligent agent analysis unit to monitor the update status; when an update abnormality is detected, a rollback operation is triggered to restore to the state before the update; the information of the intelligent agent analysis unit that has successfully updated is recorded, and an update completion list is generated;

[0146] Constructing a group collaborative management module, the group collaborative management module receives the update completion list; the group collaborative management module performs state synchronization on the intelligent agent parsing unit that has completed the update to ensure group consistency; the group collaborative management module collects load data of the intelligent agent parsing unit and calculates load distribution entropy; generates a load balancing strategy based on the load distribution entropy and performs dynamic scheduling of intelligent agent parsing tasks;

[0147] Deploy performance monitoring probes to collect optimized group performance indicators; input the optimized group performance indicators into the performance evaluation indicator system to form an evaluation optimization closed loop and complete a round of group intelligent optimization.

[0148] First, we build a performance evaluation index system, including indicators such as response time, throughput, resource utilization, and error rate. We collect indicator data in real time through performance collection probes deployed on the intelligent agent parsing unit. For example, we collect performance data such as an average response time of 200ms, a throughput of 1000QPS, a CPU utilization of 65%, and an error rate of 0.1%. We standardize the collected performance data to eliminate the impact of dimensions.

[0149] Calculate information entropy based on the standardized performance matrix. For each performance indicator, count its data distribution and calculate the information entropy value. Based on this, generate dynamic evaluation weights. For example, the response time weight is 0.3, the throughput weight is 0.3, the resource utilization weight is 0.2, and the error rate weight is 0.2. Multiply the standardized performance matrix with the weight to get the weighted indicator value. Set the benchmark threshold vector, such as the response time threshold of 300ms, the throughput threshold of 800QPS, etc. By comparing and calculating the performance scores, the weighted geometric average method is used to fuse and obtain the comprehensive score of strategy optimization.

[0150] The strategy optimization score is input into the dominance relationship analysis model, and the global optimum is determined based on the Pareto optimality criterion. When the global optimum is reached, the difference between the new and old strategies is calculated, and an incremental update data packet is generated. A hierarchical parsing network topology diagram is constructed, and a batch update sequence is designed. Incremental update data packets are transmitted between intelligent agent parsing units according to the update sequence.

[0151] Set update checkpoints in the agent parsing unit and monitor the update status. Trigger a rollback operation when an anomaly is detected. Record the information of the successfully updated agent parsing unit and generate an update completion list. The group collaborative management module receives the update completion list and performs state synchronization on the updated agents. Collect load data to calculate the load distribution entropy, generate a load balancing strategy based on this, and dynamically schedule parsing tasks.

[0152] Deploy performance monitoring probes to continuously collect optimized group performance indicators, input them into the performance evaluation indicator system for evaluation, and form an optimization closed loop. Through multiple rounds of iterative optimization, the group intelligence performance is continuously improved.

[0153] The solution of this application can:

[0154] The multi-dimensional performance evaluation index system and dynamic weight calculation method are adopted to achieve a comprehensive and accurate evaluation of the optimization strategy, and improve the scientificity and reliability of the optimization decision. The batch incremental update mechanism and state synchronization mechanism are designed to ensure the stability and consistency of the strategy update process and reduce the system operation risk. The closed-loop feedback mechanism of performance monitoring-evaluation-optimization is constructed to improve the group intelligence performance through continuous iterative optimization, and enhance the system's adaptive ability and stability.

[0155] Figure 2 FIG. 1 is a schematic diagram of the structure of a log parsing adaptive optimization system based on swarm intelligence according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0156] The first unit is used to obtain a database log to be parsed, perform feature extraction on the database log to be parsed based on a deep neural network, and generate a multi-dimensional feature matrix; construct a hierarchical parsing network according to the multi-dimensional feature matrix, and deploy a cluster of intelligent agent parsing units with self-learning ability in the hierarchical parsing network; configure a feature recognition module, a parallel computing engine, and a decision optimizer for the cluster of intelligent agent parsing units; based on the log time series characteristics of the multi-dimensional feature matrix, use a dynamic task allocation algorithm to calculate the optimal parsing solution, and allocate the parsing task to the cluster of intelligent agent parsing units;

[0157] The second unit is used for each intelligent agent parsing unit in the intelligent agent parsing unit cluster to establish a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the log pattern features are extracted through the feature recognition module, the high-concurrency parsing is performed using the parallel computing engine, and the parsing strategy is dynamically adjusted using the decision optimizer; the performance evaluation indicators in the parsing process are recorded, including parsing throughput, resource utilization and accuracy score; a multi-objective evaluation model is constructed based on the performance evaluation indicators to calculate the comprehensive ability score of each intelligent agent parsing unit; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted;

[0158] The third unit is used to receive the comprehensive ability score and the core strategy feature set fed back by the cluster of intelligent agent parsing units; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; performs performance evaluation on the optimized strategy set, and calculates the strategy optimization score; when the strategy optimization score reaches the global optimum, updates the optimized strategy set to all intelligent agent parsing units in the hierarchical parsing network, and completes a round of swarm intelligence optimization.

[0159] According to a third aspect of the embodiments of the present invention,

[0160] An electronic device is provided, comprising:

[0161] processor;

[0162] a memory for storing processor-executable instructions;

[0163] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0164] According to a fourth aspect of the embodiments of the present invention,

[0165] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0166] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The log parsing adaptive optimization method based on swarm intelligence is characterized by: include: Obtaining a database log to be analyzed, performing feature extraction on the database log to be analyzed based on a deep neural network, and generating a multi-dimensional feature matrix; Constructing a hierarchical parsing network according to the multidimensional feature matrix, and deploying a cluster of intelligent agent parsing units with self-learning ability in the hierarchical parsing network; Configuring a feature recognition module, a parallel computing engine and a decision optimizer for the agent parsing unit cluster; Based on the log time series characteristics of the multi-dimensional feature matrix, a dynamic task allocation algorithm is used to calculate the optimal parsing solution, and the parsing tasks are allocated to the agent parsing unit cluster; Each intelligent agent parsing unit in the intelligent agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the feature recognition module is used to extract log pattern features, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy; Recording performance evaluation indicators during the parsing process, including parsing throughput, resource utilization, and accuracy scores; constructing a multi-objective evaluation model based on the performance evaluation indicators, and calculating the comprehensive capability score of each of the intelligent agent parsing units; when the comprehensive capability score of a certain intelligent agent parsing unit reaches the optimal performance, extracting its core strategy feature set; Receive the comprehensive ability score and the core strategy feature set fed back by the agent analysis unit cluster; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; perform performance evaluation on the optimized strategy set and calculate the strategy optimization score; When the strategy optimization score reaches the global optimum, the optimization strategy set is updated to all intelligent agent parsing units in the hierarchical parsing network to complete a round of swarm intelligence optimization.

2. The method according to claim 1, characterized in that Each agent parsing unit in the agent parsing unit cluster establishes a group collaborative network through a high-speed data channel to share parsing status information in real time; when the agent parsing unit performs a parsing task, the feature recognition module is used to extract log pattern features, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy, including: Construct an intelligent agent state sharing network, receive log data to be parsed, input the log data into a distributed message queue, the distributed message queue is provided with a state broadcaster and a plurality of state receivers; the state broadcaster packages and broadcasts parsing state information, the parsing state information includes parsing progress parameters, resource occupancy parameters and parsing accuracy parameters; the state receiver receives and parses the parsing state information, generates a state feature matrix; calculates the system load distribution according to the state feature matrix, and outputs the load balancing parameters; A feature recognition module is configured based on the load balancing parameters, the feature recognition module includes a feature dictionary and a sample enhancement unit; the feature dictionary stores standard log patterns, the sample enhancement unit performs word segmentation and vectorization on the log data to generate a feature vector; an attention mechanism is used to extract information fragments from the feature vector; the information fragments are input into a trained neural network model, log category information and log feature information are output, and the log category information and the log feature information are sent to a parsing and scheduling module; The parsing scheduling module constructs a multi-stage pipeline parsing architecture according to the received log category information and the log feature information, wherein the multi-stage pipeline parsing architecture includes a lexical analysis unit, a grammatical analysis unit, and a semantic understanding unit; distributes parsing tasks among the lexical analysis unit, the grammatical analysis unit, and the semantic understanding unit based on a work stealing algorithm; and records the running status data of the multi-stage pipeline parsing architecture to generate a performance evaluation report; The performance evaluation report is input into a decision optimization model, which is constructed based on a deep Q network, and the log feature information, the parsing status information and the performance evaluation report are combined into a state vector; according to the state vector, an optimal control strategy is calculated through the decision optimization model, and the optimal control strategy includes parsing parallelism, batch processing parameters and algorithm selection parameters; the optimal control strategy is applied to the multi-stage pipeline parsing architecture to dynamically adjust the parsing parameters.

3. The method according to claim 2, characterized in that Inputting the performance evaluation report into a decision optimization model, wherein the decision optimization model is constructed based on a deep Q network, and the log feature information, the parsing state information and the performance evaluation report form a state vector; According to the state vector, calculating the optimal control strategy through the decision optimization model includes: Inputting the performance evaluation report into a decision optimization model, wherein the decision optimization model is constructed based on a deep Q network; performing feature preprocessing on the log feature information, the parsing state information and the performance evaluation report to generate preprocessed feature data; and combining the preprocessed feature data to construct a state vector; The state vector is input into the state representation module of the deep Q network, wherein the state representation module includes a feature extraction network and a feature fusion network; the feature extraction network performs hierarchical extraction on the feature data in the state vector to generate a feature representation sequence; the feature fusion network performs weighted fusion on the feature representation sequence to output a state representation vector; The state representation vector is input into a value assessment module, and the value assessment module calculates a Q value estimate of each candidate action based on current network parameters; selects an action with a maximum Q value as a candidate control strategy according to the Q value estimate; deploys the candidate control strategy to an analysis system for verification; collects performance feedback data of the analysis system; and calculates an execution reward of the candidate control strategy according to the performance feedback data; Constructing an experience replay buffer pool, combining the state representation vector, the candidate control strategy, the execution reward and the new state vector after execution into a state transfer sample; storing the state transfer sample into the experience replay buffer pool; randomly sampling a training batch from the experience replay buffer pool; calculating the time difference error between the target Q value and the current Q value based on the training batch; The network parameters of the deep Q network are updated according to the time difference error; the state characterization vector is re-evaluated based on the updated network parameters; an optimized control strategy is generated; the optimized control strategy is input into the analysis system; updated performance data is collected; when the performance data shows that the optimization effect does not meet the preset requirements, the strategy exploration probability is increased and the state characterization step is returned to restart the optimization; when the performance data shows that the optimization effect meets the preset requirements, the optimized control strategy is determined as the optimal control strategy.

4. The method according to claim 1, characterized in that: A multi-objective evaluation model is constructed based on the performance evaluation index, and the comprehensive ability score of each of the intelligent agent parsing units is calculated; when the comprehensive ability score of a certain intelligent agent parsing unit reaches the optimal performance, its core strategy feature set is extracted, including: The performance evaluation indicators are layered based on the principal dimension decomposition method to construct an indicator judgment matrix; the eigenvector of the indicator judgment matrix is ​​calculated to obtain the weight coefficient of each indicator; Constructing a multi-objective evaluation model for the intelligent agent analysis unit, inputting the performance evaluation index into the multi-objective evaluation model; calculating the membership of each index based on the Gaussian membership function; performing a weighted average operation on the membership and the weight coefficient to obtain a comprehensive ability score of each intelligent agent analysis unit; grouping the comprehensive ability scores using the K-means clustering algorithm to determine the intelligent agent analysis unit with the best performance; The Gaussian membership function calculation formula is as follows: ； Among them, x ij is the membership degree of indicator i to evaluation level j, x i is the actual value of index i, x j is the standard value of evaluation level j, σ is the adjustable fuzzy factor; The formula for calculating the comprehensive ability score is as follows: ； Among them, S is the comprehensive ability score, n is the number of indicators, and w i is the weight of the ith indicator, m is the number of evaluation levels, μ ij is the membership degree of the ith indicator to the jth level, v j is the quantized value of the jth level; The core operating parameters are extracted from the intelligent agent parsing unit with the best performance, and the core operating parameters include feature extraction parameters, parsing scheduling parameters and resource configuration parameters; the core operating parameters are reduced in dimension using the principal component analysis method to obtain influencing factors; a decision tree model is constructed based on the influencing factors to mine the dependencies between parameters and generate a core strategy feature set.

5. The method according to claim 1, characterized in that Construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimization strategy set through feature recombination, feedback optimization and adaptive adjustment, including: Constructing an adaptive evolution model, the adaptive evolution model includes a genetic algorithm unit and a reinforcement learning unit; inputting the core strategy feature set into the adaptive evolution model; performing feature decomposition on the core strategy feature set to extract continuous feature parameters and discrete feature rules; encoding the continuous feature parameters using real number coding, encoding the discrete feature rules using binary coding, and generating a mixed coding chromosome; Constructing an initial strategy population based on the mixed coding chromosome; collecting performance data of the initial strategy population in a target scenario, and inputting the performance data into the genetic algorithm unit; the genetic algorithm unit calculates a fitness value based on the performance data, and selects high-quality individuals according to the fitness value; performing an adaptive crossover operation on the high-quality individuals to generate crossover offspring; performing a directed mutation operation on the crossover offspring to generate a mutant offspring; The mutant offspring is input into the reinforcement learning unit; the reinforcement learning unit maps the mutant offspring into a state feature vector; a value evaluation network and a strategy generation network are constructed based on the state feature vector; environmental feedback data is collected, and a reward value is calculated based on the environmental feedback data; the value evaluation network and the strategy generation network are trained using the reward value, and the optimized strategy parameters are output; Constructing an adaptive adjustment module, inputting the optimized strategy parameters into the adaptive adjustment module; the adaptive adjustment module collects real-time performance monitoring data, and calculates performance deviation based on the real-time performance monitoring data; dynamically adjusting the learning rate and optimization direction according to the performance deviation to achieve adaptive optimization of strategy parameters; the adaptive adjustment module feeds back the adjusted strategy parameters to the genetic algorithm unit to form an optimization closed loop; Cluster analysis is performed on the strategies that have completed the adaptive evolution to identify strategy groups; the degree of complementarity of different strategies in the strategy groups is calculated; strategy combinations with synergistic effects are screened based on the degree of complementarity, and the strategy combinations are integrated into an optimized strategy set.

6. The method according to claim 5, characterized in that The genetic algorithm unit calculates a fitness value based on the performance data, selects high-quality individuals according to the fitness value, performs an adaptive crossover operation on the high-quality individuals, and generates crossover offspring; Performing a directed mutation operation on the crossover offspring to generate a mutant offspring includes: The genetic algorithm unit divides the performance data by the benchmark performance data to obtain a performance ratio matrix; constructs a dynamic weight vector, wherein the weight coefficient in the dynamic weight vector is proportional to the importance of the corresponding performance indicator; uses the product of the performance ratio matrix and the dynamic weight vector as the individual fitness value; calculates the fitness values ​​of all individuals in the population to obtain the population fitness distribution; Calculating the maximum fitness of the population and the average fitness of the population according to the population fitness distribution; constructing an adaptive selection probability based on the maximum fitness of the population and the average fitness of the population, wherein the adaptive selection probability increases as the fitness value of the individual increases; selecting high-quality individuals from the population according to the adaptive selection probability; recording the genetic structure and the corresponding fitness value of the high-quality individuals, and establishing a gene-performance mapping relationship; Analyze the gene-performance mapping relationship to identify gene segments whose contribution to performance improvement is higher than a preset performance threshold; calculate the adaptive crossover probability, which is negatively correlated with the fitness value of the individuals participating in the crossover; select the gene segments whose contribution is higher than the preset performance threshold as the crossover region; perform a crossover operation on the high-quality individuals in the crossover region based on the adaptive crossover probability to generate crossover offspring; calculate the fitness value of the crossover offspring to evaluate the effectiveness of the crossover operation; A performance gradient field based on the fitness value is constructed to determine the direction of performance improvement; an adaptive mutation probability is calculated, wherein the adaptive mutation probability decreases as the individual fitness value increases; and a directional mutation operation is performed on the crossover offspring along the direction of performance improvement to generate a mutant offspring.

7. The method according to claim 1, characterized in that The performance of the optimization strategy set is evaluated and the strategy optimization score is calculated; when the strategy optimization score reaches the global optimum, the optimization strategy set is updated to all intelligent agent parsing units in the hierarchical parsing network, and a round of swarm intelligence optimization is completed, including: Constructing a performance evaluation index system, collecting performance data of the optimization strategy set under the performance evaluation index system, and obtaining a performance data set; performing standardization processing on the performance data set to generate a standardized performance matrix; Calculating an information entropy value based on the standardized performance matrix, and generating a dynamic evaluation weight according to the information entropy value; multiplying the standardized performance matrix by the dynamic evaluation weight to obtain a weighted performance index; constructing a benchmark threshold vector, comparing the weighted performance index with the benchmark threshold vector, and calculating a performance score; fusing the performance scores using a weighted geometric mean method to generate a strategy optimization score; The strategy optimization score is input into the dominance relationship analysis model, and whether the strategy optimization score reaches the global optimum is determined based on the Pareto optimality criterion; when the strategy optimization score reaches the global optimum, the difference between the new and old strategies is calculated, and an incremental update data packet is generated; a hierarchical parsing network topology diagram is constructed, and a batch update sequence is designed based on the hierarchical parsing network topology diagram; The incremental update data packet is transmitted between the intelligent agent analysis units according to the batch update sequence; an update checkpoint is set in each intelligent agent analysis unit to monitor the update status; when an update abnormality is detected, a rollback operation is triggered to restore to the state before the update; the information of the intelligent agent analysis unit that has successfully updated is recorded, and an update completion list is generated; Constructing a group collaborative management module, the group collaborative management module receives the update completion list; the group collaborative management module performs state synchronization on the intelligent agent parsing unit that has completed the update to ensure group consistency; the group collaborative management module collects load data of the intelligent agent parsing unit and calculates load distribution entropy; generates a load balancing strategy based on the load distribution entropy and performs dynamic scheduling of intelligent agent parsing tasks; Deploy performance monitoring probes to collect optimized group performance indicators; input the optimized group performance indicators into the performance evaluation indicator system to form an evaluation optimization closed loop and complete a round of group intelligent optimization.

8. A log parsing adaptive optimization system based on swarm intelligence, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain a database log to be analyzed, perform feature extraction on the database log to be analyzed based on a deep neural network, and generate a multi-dimensional feature matrix; Constructing a hierarchical parsing network according to the multidimensional feature matrix, and deploying a cluster of intelligent agent parsing units with self-learning ability in the hierarchical parsing network; Configuring a feature recognition module, a parallel computing engine and a decision optimizer for the agent parsing unit cluster; Based on the log time series characteristics of the multi-dimensional feature matrix, a dynamic task allocation algorithm is used to calculate the optimal parsing solution, and the parsing tasks are allocated to the agent parsing unit cluster; The second unit is used for each intelligent agent parsing unit in the intelligent agent parsing unit cluster to establish a group collaborative network through a high-speed data channel to share parsing status information in real time; when the intelligent agent parsing unit performs a parsing task, the log pattern features are extracted through the feature recognition module, the parallel computing engine is used to perform high-concurrency parsing, and the decision optimizer is used to dynamically adjust the parsing strategy; Recording performance evaluation indicators during the parsing process, including parsing throughput, resource utilization, and accuracy scores; constructing a multi-objective evaluation model based on the performance evaluation indicators, and calculating the comprehensive capability score of each of the intelligent agent parsing units; when the comprehensive capability score of a certain intelligent agent parsing unit reaches the optimal performance, extracting its core strategy feature set; The third unit is used to receive the comprehensive ability score and the core strategy feature set fed back by the cluster of intelligent agent analysis units; construct an adaptive evolution model, and input the core strategy feature set into the adaptive evolution model; the adaptive evolution model integrates genetic algorithm and reinforcement learning mechanism, and generates an optimized strategy set through feature recombination, feedback optimization and adaptive adjustment; and performs performance evaluation on the optimized strategy set to calculate the strategy optimization score; When the strategy optimization score reaches the global optimum, the optimization strategy set is updated to all intelligent agent parsing units in the hierarchical parsing network to complete a round of swarm intelligence optimization.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images