Equipment log processing method for rail transit intelligent operation and maintenance system

Abstract

The invention discloses an equipment log processing method for a rail transit intelligent operation and maintenance system. A system consisting of an equipment log acquisition module, an intelligent analysis module, a data storage module and a visual display module is adopted to realize the monitoring and log processing functions of equipment operation state information. An equipment log collection module monitors a server specified log catalog in real time, and when adding or modifying occurs in the catalog, log incremental data are extracted immediately; the intelligent analysis module performs cleaning, feature extraction, log clustering, template extraction, template registration and exception marking on the collected log incremental data to realize conversion from an unstructured log to structured data; the data storage module adopts Apache Dores as a storage carrier to carry out persistent storage on the structured log data; and the visual display module displays the structured log data in a view form. According to the method, the log utilization value is improved, log standardization is realized, and unified management and global analysis are supported.

Description

TECHNICAL FIELD

[0001] The present application relates to rail transit equipment operation and maintenance equipment log processing technology, in particular to a device log processing method for a rail transit intelligent operation and maintenance system. BACKGROUND

[0002] In the rail transit intelligent operation and maintenance system, the device log is the core data for recording the device running state, operation behavior and abnormal information, and its processing quality directly affects the operation and maintenance efficiency and risk control ability. The existing log processing scheme has many deficiencies: first, the acquisition link adopts a timing polling method, which cannot capture log incremental data in real time, resulting in late discovery of abnormal information; second, the format compatibility is poor, and the support for non-traditional log formats such as.zip and.xlsx is insufficient, which is easy to cause data acquisition omission; third, the parsing link highly depends on manual configuration templates, and the processing efficiency of unstructured logs is low, and when the log format changes, the template update cycle is long and the adaptability is poor; fourth, after mass log storage, there is a lack of efficient indexing mechanism, and the query analysis is time-consuming, which is difficult to quickly support operation and maintenance decision-making. SUMMARY

[0003] In view of the problems existing in the prior art, the present application provides a device log processing method for a rail transit intelligent operation and maintenance system. The method has a device log processing scheme with real-time acquisition, intelligent analysis, efficient storage and intuitive display capabilities to improve the utilization value of log data and reduce operation and maintenance costs.

[0004] The technical scheme adopted by the present application is: a device log processing method for a rail transit intelligent operation and maintenance system adopts a device log processing system composed of a device log acquisition module, an intelligent analysis module, a data storage module and a visual display module, realizes the monitoring of device running state information and log processing functions, and the steps are as follows: S1, device log acquisition: the device log acquisition module monitors the server specified log directory in real time, and when new or modified occurs in the log directory, the log incremental data is immediately extracted.

[0005] S2, intelligent analysis: the intelligent analysis module performs cleaning, feature extraction, log clustering, template extraction, template registration and abnormal marking on the collected log incremental data, and realizes the conversion of unstructured log to structured data.

[0006] First, spaces, special symbols, and duplicate data are removed from the collected incremental log data. Then, the text data is converted into semantic vectors. The semantic vectors are clustered to group semantic vectors with similar structures into the same category. Log structure features are extracted from the clustering results to generate corresponding parsing templates. The generated parsing templates are added to the template registry. Log data that cannot be clustered during the clustering process is marked as unknown log types and left for manual review before further processing.

[0007] S3. Data storage: The data storage module uses Apache Doris as the storage medium to persistently store the obtained structured log data.

[0008] S4. Visualization: The visualization module displays structured log data in a view form through a UI interface.

[0009] The data flow process of the device log processing system is as follows: After the device log collection module enters the system, it first generates incremental log data through the incremental extraction unit. The incremental log data is then transmitted to the intelligent parsing module, where it passes through the data cleaning unit, feature transformation unit, clustering processing unit, template generation unit, template management unit, and anomaly handling unit in sequence to complete the structured transformation, template matching, and anomaly marking of the logs. The parsed structured log data is then transmitted to the data storage module for persistent storage via an interface. The visualization module accesses the structured log data through the data storage module and presents it to the user in the form of dashboards, statistical views, and classification views, enabling real-time monitoring and analysis.

[0010] The beneficial effects of this invention are: 1. Enhance the value of log utilization: By collecting, parsing and analyzing massive amounts of logs in real time, we can deeply mine the equipment operation information contained in the log data, transforming passive reading into proactive monitoring and prediction, and significantly improving log utilization.

[0011] 2. Achieve log standardization: By using a unified data model and parsing rules, effectively address log format differences and lay the foundation for subsequent integration and analysis.

[0012] 3. Supports unified management and global analysis: Utilizes Apache Doris to centrally manage all log data, providing a unified view and correlation analysis capabilities across devices and the entire process, making it easier to discover systemic problems and potential risk chains. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating the data flow process of the device log processing system of the present invention. Figure 2 This is a flowchart of the steps of the device log processing method of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] like Figure 1 , Figure 2 As shown, the equipment log processing method for intelligent operation and maintenance systems of rail transit adopts an equipment log processing system composed of an equipment log acquisition module, an intelligent parsing module, a data storage module, and a visualization display module to monitor the equipment operating status information; the modules are connected in sequence to work together to realize the log processing function.

[0016] The device log collection module is used to monitor a specified log directory on the server in real time, capture file addition or modification events and extract incremental data, and supports the collection of multiple mainstream log formats.

[0017] The intelligent parsing module is connected to the device log collection module and is used to clean, extract features, cluster logs, extract templates, register templates, and mark anomalies in the collected incremental log data, thereby converting unstructured logs into structured data.

[0018] The data storage module is connected to the intelligent parsing module and uses an Apache Doris distributed cluster to persistently store structured log data and synchronously generate data indexes.

[0019] The visualization module is connected to the data storage module and is used to display structured log data in a multi-dimensional view through a UI interface, supporting interactive operation.

[0020] The device log acquisition module monitors the specified log directory on the server in real time. When new or modified data is added to the log directory, incremental log data is extracted immediately.

[0021] The device log acquisition module includes a monitoring unit, a format parsing unit, and an incremental extraction unit. The monitoring unit uses the inotify mechanism to listen for file change events in a specified log directory, including file creation, modification, and deletion. The format parsing unit has built-in parsers for various formats: for .zip format files, it uses a decompression tool to extract log files; for .xlsx format files, it uses the POI component to read table data; and for .log, .txt, and .xml formats, it directly parses the text content. The incremental extraction unit extracts only the newly added or modified data from the files based on the events captured by the monitoring unit, avoiding the resource waste caused by reading the entire file.

[0022] Employing a file system monitoring mechanism, the system monitors a specified log directory on the server in real time. By listening for inode change events in the specified log directory, it immediately extracts incremental log data when files are added or modified within the directory. This real-time capture of file addition and modification operations ensures the real-time nature of incremental log data extraction, with a collection latency of no more than 1 second. It also supports log collection in multiple mainstream formats, including .log, .txt, .xml, .zip, and .xlsx. For .zip compressed log files, it automatically performs decompression and extracts the internal log data; for .xlsx log files, it extracts log content by reading specified rows and columns of a table, avoiding collection omissions due to format incompatibility.

[0023] The intelligent parsing module includes a data cleaning unit, a feature transformation unit, a clustering unit, a template generation unit, a template management unit, and an anomaly handling unit. The data cleaning unit first removes spaces, special characters, and duplicate data from the collected incremental log data. Duplicate data cleaning uses an MD5 hash-based deduplication method, calculating the MD5 value for each log entry and deleting duplicate data with the same MD5 value as already stored log entries. The feature transformation unit converts the cleaned text data into semantic vectors using the TF-IDF algorithm; that is, it takes the cleaned log data, selects an appropriate transformation method based on the log complexity, extracts keyword features, and generates semantic vectors, ensuring the accuracy of feature extraction. The clustering unit clusters the semantic vectors, grouping semantic vectors with similar structures into the same category. The template generation unit extracts log structural features from the clustering results and generates corresponding parsing templates. The template management unit adds the generated new parsing templates to the template registry. The anomaly handling unit marks log data that cannot be clustered during the clustering process as unknown log types, awaiting manual review before further processing.

[0024] The data flow process of the device log processing system is as follows: After the device log collection module enters the system, it first generates incremental log data through the incremental extraction unit. The incremental log data is then transmitted to the intelligent parsing module, where it passes through the data cleaning unit, feature transformation unit, clustering processing unit, template generation unit, template management unit, and anomaly handling unit in sequence to complete the structured transformation, template matching, and anomaly marking of the logs. The parsed structured log data is then transmitted to the data storage module for persistent storage via an interface. The visualization module accesses the structured log data through the data storage module and presents it to the user in the form of dashboards, statistical views, and classification views, enabling real-time monitoring and analysis.

[0025] Example of a feature transformation unit converting textual data into semantic vectors: In this system, even after cleaning, the logs remain unstructured text with diverse sentence structures and significant content variations, making it difficult to determine semantic similarity through direct text comparison. For example, the following log content: 2025-11-10 10:12:45 Device-A01 temperature too high, current value: 85℃; 2025-11-10 10:12:50 Device-A01 Temperature abnormality, value 85℃.

[0026] While the words and sentence structures may appear different on the surface, they actually reflect the same type of alert information. Directly using them for clustering or modeling would prevent logs with similar semantics from being classified into the same category.

[0027] To achieve structured representation, the feature transformation unit uses the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm to vectorize the log text. The specific implementation steps are as follows: 1. Word segmentation and term statistics: The system segments the log text (e.g., "temperature is too high, current value is 85℃") into words, obtains the term sequence, and counts the number of times each term appears in the log and the total number of words in the log, providing basic data for subsequent weight calculation.

[0028] 2. Calculate Term Frequency (TF): Term frequency is expressed by the formula TF = n t Calculate / N, where n t N is the number of times a term appears in the log, and N is the total number of words in the log.

[0029] 3. Calculate the Inverse Document Frequency (IDF): The inverse document frequency is calculated as IDF = ln(D / (d)). t Calculate using + 1), where D is the total number of logs in the system, and d t This represents the number of log entries containing this term.

[0030] 4. Calculate TF-IDF weights: Weight = TF × IDF. The system calculates the weights for all terms in this batch sequentially, forming the weight vector corresponding to the log.

[0031] 5. Vectorization transformation: Under a unified feature space, each log is mapped into a numerical vector of fixed dimensions, which facilitates subsequent calculation and analysis.

[0032] Log vector A: [0.23, -0.14, 0.65, …]; Log vector B: [0.21, -0.12, 0.68, …].

[0033] In the vector space, the system calculates the distance between log vectors using methods such as cosine similarity. Based on experimental and business rules, a threshold can be set to determine whether logs belong to the same category. When the distance between two log vectors is ≤ 0.1, they are determined to be logs of the same type; When the distance between two log vectors is ≥ 0.3, they are determined to be logs of different classes. For logs with a distance between 0.1 and 0.3, further judgment can be made by combining rules such as log type or alarm level.

[0034] Through the above processing, each log is converted into a computable numerical vector. Logs with similar semantics are close in distance in the vector space, thus providing reliable input for subsequent clustering, template extraction and alarm analysis.

[0035] This method uses the K-Means algorithm to cluster the obtained semantic vectors. The number of clusters is determined by the elbow rule. First, the silhouette coefficients under different numbers of clusters are calculated, and the number of clusters corresponding to the largest silhouette coefficient is selected as the optimal number of clusters. The K-Means algorithm is an unsupervised learning method based on distance metrics. Its core is to divide the data into K categories, so that the similarity of samples in the same category is maximized and the difference between samples in different categories is maximized.

[0036] To determine the optimal number of clusters, this method employs the Elbow Method for parameter optimization. The specific steps include: setting different cluster size values ​​K (e.g., K=2, 3, 4, ..., 20) for the same batch of semantic vector log samples; for each K value, executing the K-Means algorithm and calculating the sum of squared errors (SSE) from all sample points to their corresponding cluster centers; as the K value increases, the sum of squared errors SSE gradually decreases, but the rate of decrease begins to slow significantly at a certain point, forming an elbow-like inflection point. The K value corresponding to this inflection point is the optimal number of clusters.

[0037] Example explanation: For a set of log vectors, the sum of squared errors (SSE) for different K values ​​is shown in Table 1. Table 1 SSE results for different K values

[0038] Based on the above results, it can be seen that when K=5, the decrease in the sum of squared errors (SSE) slows down significantly. At this point, K=5 is selected as the optimal number of clusters to balance the compactness and generalization ability of the clusters.

[0039] After determining the optimal value of K, the system performs a K-Means clustering process, which includes randomly initializing K cluster centers; calculating the distance between each log vector and each cluster center, and assigning it to the category corresponding to the nearest cluster center; recalculating the center point of each category, and repeating the above steps until the cluster centers converge or the preset number of iterations is reached; after clustering is completed, logs with similar semantics are grouped into the same category, providing a basis for subsequent log template extraction, alarm analysis and anomaly detection.

[0040] The template extraction in this method automatically extracts fixed and variable fields from the logs in each clustering result, generating a parsing template for the corresponding category. The parsing template for the log's category is defined. For example, for a log with the [Time][Device ID] running status: Normal, the extracted template would be: [Time field][Device ID field] Running status: [Status field]. Using this template, subsequent logs of the same type can be directly matched and parsed, achieving automated structured transformation.

[0041] The template registration in this method automatically adds the newly generated parsing template to the parser's template library, establishing a mapping relationship between templates and log categories. Subsequent collection of logs of the same type can directly match the corresponding parsing template for structured parsing, eliminating the need for repeated clustering and improving parsing efficiency.

[0042] The anomaly labeling in this method addresses low-frequency log data that occurs during clustering. Specifically, when a log entry appears less than a preset threshold (e.g., 10 times) within a given period or batch of data, the system labels it as an "unknown log type" and stores it in the anomaly log database. For stored anomaly logs, operations and maintenance personnel can manually review them, supplement parsing rules, or optimize clustering model parameters before reprocessing, thereby ensuring the completeness and accuracy of log parsing.

[0043] To achieve a unified and structured conversion of log formats during intelligent parsing, this method proposes a Unified Log Data Model (ULDM). The ULDM model maps log information from different sources and in different formats to a unified field structure, including standard fields such as timestamp, device number, event level, event code, and event description.

[0044] The following log parsing mapping table 2 provides examples of raw logs and structured logs: Table 2 Log Parsing Mapping Table

[0045] Table 2 lists typical examples of multi-format log mapping, demonstrating the unified structured mapping results of different log formats in ULDM. The ULDM model achieves semantic alignment of logs across systems, enabling logs from different vendors and systems to be directly statistically analyzed and visualized under a unified structure.

[0046] The data storage module uses Apache Doris as the storage medium to persistently store the obtained structured log data.

[0047] The visualization module presents structured log data in a view format through a UI interface. This includes a device operation status dashboard displaying the percentage of devices in normal / abnormal states; an alarm information statistics view showing the number of alarms by alarm level and type; and a log category statistics view displaying the quantity distribution by log category. The interface supports interactive operation, allowing maintenance personnel to quickly locate target logs using filtering conditions such as time range and device ID, view log details through operations, and save log data as a local Excel file using the export function.

Claims

1. A device log processing method for an intelligent operation and maintenance system for rail transit, characterized in that, The device log processing system composed of a device log collection module, an intelligent analysis module, a data storage module and a visual display module is adopted to realize the monitoring and log processing functions of the device running state information, and the steps are as follows: S1, device log collection: the device log collection module monitors the server specified log directory in real time, and immediately extracts the log incremental data when the log directory is newly added or modified; S2, intelligent analysis: the intelligent analysis module cleans, extracts features, clusters, extracts templates, registers templates and marks abnormalities for the collected log incremental data, realizing the conversion from unstructured log to structured data; Firstly, the collected log incremental data is removed from the space, special symbols and repeated data; secondly, the text data is converted into semantic vectors; The semantic vectors are clustered, and the similar semantic vectors are classified into the same category; The log structure features are extracted from the clustering results, and the corresponding analysis templates are generated; The generated new analysis templates are added to the template registration table; The log data that cannot be clustered in the clustering process is marked as unknown log type and left for manual review and processing; S3, data storage: the data storage module uses Apache Doris as the storage carrier to persistently store the structured log data obtained; S4, visual display: the visual display module displays the structured log data in the form of view through the UI interface.

2. The device log processing method for the rail transit intelligent operation and maintenance system according to claim 1, characterized in that, In step S1, the file system monitoring mechanism is adopted to listen to the inode change event of the specified log directory, capture the file addition and modification operation in real time, and ensure that the log data extraction is real-time, with a delay of not more than 1 second.

3. The device log processing method for the rail transit intelligent operation and maintenance system according to claim 1, characterized in that, In step S2, the TF-IDF algorithm is used to convert the cleaned log data into semantic vectors.

4. The device log processing method for the rail transit intelligent operation and maintenance system according to claim 1, characterized in that, In step S2, the K-Means algorithm is used to cluster the obtained semantic vectors, and the number of clusters is determined by the elbow rule. First, calculate the silhouette coefficient under different cluster numbers, and select the cluster number corresponding to the maximum silhouette coefficient as the optimal cluster number.

5. The device log processing method for an intelligent rail transit operation and maintenance system according to any one of claims 1 to 4, characterized in that, The data flow process of the device log processing system is as follows: After the device log collection module enters the system, the incremental log data is first generated by the incremental extraction unit; the incremental log data is transmitted to the intelligent analysis module, and sequentially passes through the data cleaning unit, the feature conversion unit, the clustering processing unit, the template generation unit, the template management unit and the abnormal processing unit, to complete the structured conversion, template matching and abnormal marking of the log; the analyzed structured log data is transmitted to the data storage module through the interface for persistent storage; The visual display module presents the structured log data to the user in the form of dashboard, statistical view and classification view through the data storage module, realizing real-time monitoring and analysis.

Images