Rare earth industry-oriented mass log data intelligent association analysis method and system

Abstract

The invention belongs to the technical field of log analysis and intelligent operation and maintenance, and discloses a rare earth industry-oriented mass log data intelligent association analysis method and system. Aiming at the problems that traditional log analysis depends on manpower, the efficiency is low and cross-system association is difficult to discover, heterogeneous log data generated by multiple systems are converged, and a mode self-learning algorithm is adopted to automatically identify a log format and extract structured information; performing association mining on the log events based on time sequence co-occurrence analysis and semantic similarity analysis; and constructing a log event association graph, and automatically locking a fault source from massive association events by adopting a causal reasoning algorithm. According to the method, the fault positioning time is shortened from the hour level to the minute level, the root cause locking accuracy is 95% or above, and the method is suitable for operation and maintenance monitoring and fault diagnosis scenes of a rare earth enterprise informatization system.

Description

Technical Field

[0001] This invention belongs to the field of log analysis and intelligent operation and maintenance technology, specifically involving a massive log intelligent correlation analysis solution for rare earth enterprise information systems. It can be widely applied to the operation and maintenance monitoring, fault diagnosis and performance optimization of rare earth enterprise information systems such as manufacturing execution platforms, resource planning platforms, and supplier collaboration platforms. Background Technology

[0002] Rare earth enterprise information systems encompass multiple business platforms, including manufacturing execution management, enterprise resource planning, and supplier relationship management, generating massive amounts of log data daily. When system failures occur, maintenance personnel need to sift through logs from multiple systems to find the cause of the problem.

[0003] Existing log analysis methods have the following shortcomings: 1. Manual analysis is inefficient: Operations and maintenance personnel need to log in to multiple systems to view logs, manually search and correlate them, and fault location can take hours or even days; 2. Heterogeneous log formats: Different systems and versions have different log formats, lacking a unified means of parsing and querying; 3. Difficulty in cross-system correlation: Faults often propagate across multiple systems, and traditional tools struggle to discover cross-system log correlations; 4. Inaccurate root cause identification: The massive logs contain a lot of irrelevant information, making it difficult for humans to accurately identify the root cause of the failure. 5. Difficulty in knowledge accumulation: Troubleshooting experience depends on individuals and is difficult to form a reusable knowledge base.

[0004] The aforementioned limitations prevent existing technologies from achieving an integrated design of "intelligent analysis + correlation analysis + root cause identification," making it difficult to meet the needs of efficient operation and maintenance of rare earth enterprise information systems. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of traditional log analysis methods, such as low efficiency, difficulty in cross-system correlation, and inaccurate root cause identification, this invention proposes an intelligent correlation analysis solution for massive logs in the rare earth industry. The goal is to reduce fault location time from hours to minutes, achieve a root cause identification accuracy of over 95%, and improve operation and maintenance efficiency by 80%.

[0006] (II) Technical Solution The core idea of ​​this invention is to construct an integrated log analysis system that combines "intelligent parsing + two-dimensional association + graph reasoning". The specific technical path is as follows: 1. Multi-source log aggregation and parsing Log aggregation architecture: Deploy log collection agents on servers of various business platforms to aggregate application logs in real time; A distributed message queue is used as the log transmission channel to ensure that logs are not lost; Logs are written to the full-text search cluster for storage and indexing; Intelligent log parsing: Template learning is performed using a log-based self-learning algorithm. Organize the logs using a prefix tree, grouping similar logs into the same branch; Extract the log template (fixed part) and dynamic parameters (variable part); The template library is dynamically updated, and new log formats are automatically recognized. The parsing results are structured: { "timestamp": "2024-03-15T10:30:45.123Z", "level": "ERROR", "module": "OrderService", "template": "Failed to connect to database<*>", "params": ["mysql-master:3306"], "raw": "Failed to connect to mysql-master database: 3306" } 2. Time series correlation analysis Co-occurrence frequency calculation: Define the time window W (default 5 minutes); Count the number of times event A and event B co-occur within the window, N(A,B); Calculate the conditional probability P(B|A) = N(A,B) / N(A); Calculate the correlation strength Corr(A,B) = P(B|A) / P(B); Chronological sequence analysis: Analyze the time difference distribution between event A and event B; If A usually occurs before B (the average time difference is greater than 0), then establish a directed edge from A to B. The edge weight is the correlation strength Corr(A,B); Cross-system association: Synchronization using the Network Time Protocol ensures time consistency across all systems. It supports time-series correlation analysis across manufacturing execution, resource planning, and supplier collaboration platforms.

[0007] 3. Semantic Relationship Analysis Log vectorization: A pre-trained log semantic understanding model (based on a deep language model and fine-tuned on large-scale log data) is used. Convert the log template into a 768-dimensional vector representation; Vector caching allows vectors to be reused with the same template, improving efficiency. Semantic similarity calculation: Calculate the vector cosine similarity of event pairs: Sim(A,B) = cos(vec(A), vec(B)); Semantic association edges are established when the similarity exceeds 0.8; Semantic clustering: Density clustering algorithm is used to cluster event vectors; Events within the same cluster are considered as different manifestations of the same problem.

[0008] 4. Association map construction and root cause identification Map construction: Node: Log event (template + parameters); Edges: Temporally related edges (directed, with time difference weights) and semantically related edges (undirected, with similarity weights). Storage: Graph database, supporting graph query language; Root Cause Ranking Locking Algorithm: Input: Alarm event node A; Initialization: The root cause probability of A is 1, and the probability of other nodes is 0; Iteration: Probability is backpropagated along temporally related edges, with probabilities allocated according to edge weights; Convergence: Iterate until the probability distribution stabilizes; Output: The three nodes with the highest root cause probability are considered as suspected root causes; Faulty link reconstruction: Starting from the root cause node, traverse forward along the temporal association edges to the alarm node; Construct a complete fault propagation chain.

[0009] 5. Visualization and Report Generation Graph Visualization: Atlas rendering is achieved based on a force-guided layout algorithm; Supports node filtering (by system, by level), path highlighting, and timeline playback; Supports exporting to image or document formats; Diagnostic report: Automatically generate fault diagnosis reports, covering: Fault Overview: Alarm time, system impact, and service impact; Root cause analysis: suspected root cause, confidence level, chain of evidence; Fault chain: The complete propagation path from root cause to alarm; Recommendations for handling the situation: Recommendations based on historical cases.

[0010] (III) Beneficial Effects 1. Significantly improved fault location efficiency: reduced from hours to minutes, improving operation and maintenance efficiency by 80%; 2. High accuracy in root cause identification: The root cause ranking algorithm has an accuracy rate of over 95%, reducing false positives; 3. Strong cross-system integration capabilities: Supports unified analysis of logs from multiple platforms, including manufacturing execution, resource planning, and supplier collaboration; 4. Knowledge can be accumulated: Fault cases and handling experience are automatically accumulated to form a knowledge base; 5. Visual and intuitive: Visualized diagrams make the fault propagation path clear at a glance, lowering the barrier to understanding. IV. Description of the attached drawings Appendix Figure 1 This is a schematic diagram of the architecture of a massive log intelligent correlation analysis system. V. Detailed Implementation Methods The following example, taking the order synchronization failure between the manufacturing execution platform and the resource planning platform of a rare earth enterprise, illustrates the specific implementation process of this invention in detail: (a) Implementation preparation 1. Log aggregation environment configuration: Manufacturing execution platform: 3 application servers, with a log collection agent deployed to aggregate application logs; Resource planning platform: 2 application servers, deploying a log collection agent to aggregate application logs; Middleware: message queues, caches, databases, and aggregated middleware logs; Message queue cluster: 3 nodes, serving as a log transmission channel; Full-text search cluster: 3 nodes, storing parsed structured logs; 2. Analyze the environment configuration: Graph database: Single-node deployment, 16GB RAM, stores relational graphs; Analysis server: 32-core processor, 64GB RAM, running association analysis and root cause locking algorithms; Log semantic model: a pre-trained model, deployed on a graphics processing unit server for vectorization; 3. Log Sample: Log volume: Approximately 500,000 log entries / day for the Manufacturing Execution Platform and approximately 300,000 log entries / day for the Resource Planning Platform; Log format: The manufacturing execution platform uses the standard log framework format, while the resource planning platform uses a custom format.

[0013] (II) Implementation Steps 1. Log aggregation and parsing: Configure log collection agent rules to aggregate logs in real time and send them to the message queue; The log processor consumes message queue data and calls a pattern self-learning algorithm to learn templates. Initial learning: Analyzing a week's historical logs, 127 templates for the Manufacturing Execution Platform and 89 templates for the Resource Planning Platform were extracted; After parsing, the data is written to the full-text search cluster to create a time index.

[0014] 2. Execution of association analysis: Temporal correlation analysis: Set a 5-minute time window; Analysis revealed a high correlation between the "order creation successful" event on the Manufacturing Execution Platform and the "order receipt" event on the Resource Planning Platform (Corr=15.3). Analysis revealed that the "database connection timeout" event on the manufacturing execution platform typically precedes the "order synchronization failure" event (average time difference of 2.3 seconds). Semantic association analysis: Vectorize all templates; The semantic similarity between the "database connection timeout" error of the manufacturing execution platform and the "database connection failure" error of the resource planning platform was found to be 0.92. Graph construction: The analysis results are written into the graph database to construct an association graph containing 216 nodes and 1847 edges.

[0015] 3. Root cause identification: Fault scenario: At 10:30 a.m. on a certain day, the resource planning platform issued an alarm stating that "the order synchronization failure rate exceeds 50%"; Root cause analysis: Starting with the "order synchronization failure" event, run the root cause ranking algorithm; The algorithm traverses backwards along the temporally related edges to find the propagation path: Database master disk alarm → Manufacturing Execution Platform database connection timeout → Manufacturing Execution Platform order query failure → Order synchronization failure Root cause probability: Database master disk alarm (0.87), manufacturing execution platform database connection pool exhaustion (0.09), network jitter (0.04). The primary cause of the problem is a disk alarm on the primary database, with a confidence level of 87%.

[0016] 4. Visualization and Reporting: Visualization of the fault propagation chain: Highlighting the complete path from database alarm to order synchronization failure; Generate diagnostic report: Fault Overview: At 10:30 AM on a certain day, the order synchronization failure rate reached 52%, affecting the Manufacturing Execution and Resource Planning platform; Root cause analysis: The primary database disk utilization reached 95%, causing query timeouts; Recommended solutions: Expand the database disk or clean up historical data.

[0017] (III) Verification of Implementation Results 1. Positioning efficiency verification: The time from alarm to root cause identification for this fault was 3 minutes and 42 seconds. Traditional manual analysis is estimated to take 2 to 4 hours; Efficiency improvement: more than 30 times; 2. Accuracy verification: Root cause verification: The operations and maintenance personnel confirmed that the database disk usage was indeed the root cause, and the cause was accurately identified. Historical validation: Retrospective analysis of 15 failures over the past three months showed that the root cause was accurately identified 14 times, achieving an accuracy rate of 93.3%. 3. Business Performance Verification: Fault recovery time (average recovery time): reduced from an average of 4 hours to 30 minutes; Reduced maintenance manpower: Reduced manual log analysis workload by 80%; 4. Knowledge Accumulation Verification: Fault Case Library: Automatically accumulates 32 fault cases; Reusable handling suggestions: Similar faults can be automatically recommended based on historical handling solutions.

[0018] This implementation case fully verifies the feasibility and advancement of the present invention. It achieves efficient fault diagnosis of rare earth enterprise information systems through intelligent log analysis. The parsing rules and analysis parameters can be adjusted according to the characteristics of different systems and logs, and it has good versatility and scalability.

Claims

1. A method for intelligent correlation analysis of massive log data in the rare earth industry, characterized in that, Includes the following steps: a. Multi-source log aggregation and parsing: Application logs, middleware logs, and database logs from systems such as the manufacturing execution platform, resource planning platform, and supplier collaboration platform are aggregated through a log collection agent. A log pattern self-learning algorithm is used to automatically learn log templates, parsing unstructured logs into structured events (timestamp, severity level, functional module, event template, dynamic parameters). The log template learning accuracy reaches over 95%, and the automatic recognition time for new log formats is less than 5 minutes. The parsed structured logs are written to a full-text search cluster, establishing time and content indexes. b. Time-series correlation analysis: Define a time window (default 5 minutes) and count the co-occurrence frequency of different log events within the window; calculate the time-series correlation strength of event pairs: Corr(A,B) = P(B|A) / P(B), and consider a time-series correlation to exist when Corr exceeds a threshold (default 2.0); use a sliding window to analyze the temporal relationship of events and construct event time-series links; support cross-system time-series correlation, and ensure time-series accuracy through a unified time base (network time synchronization, error less than 10 milliseconds); c. Semantic association analysis: A pre-trained log semantic understanding model is used to vectorize log events (768-dimensional vectors); the vector cosine similarity of event pairs is calculated, and semantic association is considered to exist when the similarity exceeds 0.8; semantic clustering of log events is constructed to group similar events into the same event cluster; Supports cross-system semantic association, identifying logs from different systems that describe the same problem; d. Association Graph Construction and Root Cause Identification: Log events are used as nodes, and temporal and semantic associations are used as edges to construct an event association graph. The graph is stored in a graph database and supports complex association queries. When a fault occurs, starting from the alarm event, an improved network ranking algorithm (root cause ranking algorithm) is used to traverse backwards along the association edges to calculate the root cause probability of each node. The three events with the highest root cause probability are output as suspected root causes, and the complete fault propagation chain is displayed. e. Visualization and Report Generation: Provides an interactive visualization interface for the relational graph, supporting node expansion, path highlighting, and timeline playback; automatically generates fault diagnosis reports covering root cause analysis, impact scope, and handling recommendations. Its features include: a. Log Aggregation Module: Supports multiple access methods such as system log protocols, file collection, and message queues, with a log throughput exceeding 100,000 records per second; b. Log Parsing Module: Integrates parsing algorithms such as log pattern self-learning, achieving a parsing speed of over 50,000 records per second, with an automatically updated template library; c. Relationship Analysis Module: Includes a time-series analysis engine and a semantic analysis engine, with a single relationship analysis taking less than 10 seconds; d. Graph Storage Module: Employs a graph database cluster, supporting graph storage and querying of hundreds of millions of nodes and billions of edges; e. Root Cause Locating Module: Implements a root cause ranking algorithm, with root cause locating taking less than 30 seconds and an accuracy rate exceeding 95%; f. Visualization Module: Implements graph visualization based on a graphics rendering library, supporting smooth rendering of thousands of nodes. The system is characterized in that, in step 2, the time-series correlation analysis supports configuring different time window granularities (1 minute / 5 minutes / 15 minutes) to adapt to scenarios with different fault propagation speeds. It is also characterized in that, in step 4, the root cause ranking algorithm introduces domain knowledge weights, assigning higher prior probabilities to known common root cause types (such as database connection pool exhaustion and memory overflow). Furthermore, the system supports integration with an alarm system, automatically initiating root cause analysis when an alarm is triggered, and automatically associating the analysis results with the alarm work order.

Images