Rail transit equipment log correlation analysis platform and correlation analysis method

Abstract

A correlation analysis platform and a correlation analysis method for rail transit equipment logs are characterized in that a log management module analyzes uploaded logs and stores the analyzed logs in a database, and a data interaction module accesses a data processing engine through a data processing engine interface; the data processing engine compiles a data query statement to perform query and interactive analysis on logs in the database, an analysis result is visually displayed, a log analysis module configures different analysis rules through the analysis rule configuration interface, the logs in the database are automatically analyzed according to the analysis rules, and the analysis result is displayed according to the analysis rules. And generating a log analysis result. According to the invention, more efficient and more-dimensional log analysis means and method are provided, the log utilization rate and analysis efficiency are improved, and the purposes of more efficient equipment guarantee and maintenance and more stable equipment operation are achieved.

Description

Technical Field

[0001] The invention relates to a correlation analysis platform and a correlation analysis method for rail transit equipment logs. Background Art

[0002] Rail transit equipment is an important component and cornerstone of the rail transit system. Ensuring and maintaining its normal operation is crucial to providing continuous and reliable rail transit travel services. The guarantee and maintenance of equipment depends on the acquisition and analysis of key information such as the main parameters of equipment operation, failure conditions, and equipment operation records. Equipment logs are an important way to understand the equipment operation status and analyze the causes of equipment failures.

[0003] Currently, in the rail transit industry, the traditional way of obtaining and analyzing logs is generally to upload logs to an FTP server (File Transfer Protocol server) or other storage through event or scheduled task triggering. Maintenance and analysis personnel then obtain logs from the storage and analyze them through manual viewing or log parsing software, or parse and display a specific type of log through a log template for log analysts to review and use.

[0004] The above solution can basically meet the protection and maintenance needs of the equipment. However, since the log files are independent, massive and complex, it is difficult to manually correlate the contents of multiple log files and analyze the status changes of the equipment over a long period of time. It is difficult to grasp the changes in equipment characteristics and the laws of failure occurrence, and then establish targeted and systematic equipment maintenance and protection plans, or even equipment selection plans.

[0005] The statements herein merely provide background art related to the present invention and do not necessarily constitute prior art. Summary of the invention

[0006] The purpose of the present invention is to provide a correlation analysis platform and correlation analysis method for rail transit equipment logs, to provide more efficient and multi-dimensional log analysis means and methods, to improve the utilization rate of logs and the efficiency of analysis, and to achieve more efficient equipment protection and maintenance and more stable equipment operation.

[0007] In order to achieve the above object, the present invention provides a rail transit equipment log correlation analysis platform, comprising:

[0008] Log management module, used to parse uploaded logs;

[0009] Database, used to store parsed logs;

[0010] A data interaction module provides a data processing engine interface. The data interaction module accesses the data processing engine through the data processing engine interface and queries, interactively analyzes and visualizes the logs in the database through the data query statements of the data processing engine.

[0011] The log analysis module provides a parsing rule configuration interface. The log analysis module configures different parsing rules through the parsing rule configuration interface, automatically analyzes the logs in the database according to the parsing rules, and generates log analysis results.

[0012] The log management module provides a log upload interface, and the log upload interface is used to upload logs to the log management module.

[0013] The database at least includes: a relational database, a non-relational database, and an analytical database.

[0014] The data interaction module adopts Zeppelin server.

[0015] The data processing engine at least includes: spark, hive, and the languages ​​supported by the data processing engine at least include: Scala, Python, Spark SQL, Hive, Markdown, and Shell.

[0016] The log analysis module provides an analysis result output interface, through which the log analysis result is output.

[0017] The present invention also provides a method for correlation analysis of rail transit equipment logs, comprising:

[0018] The log management module parses the uploaded logs and stores the parsed logs in the database;

[0019] The data interaction module accesses the data processing engine through the data processing engine interface, writes data query statements through the data processing engine to query and interactively analyze the logs in the database, and visualizes the analysis results;

[0020] The log analysis module configures different analysis rules through the analysis rule configuration interface, automatically analyzes the logs in the database according to the analysis rules, and generates log analysis results.

[0021] The log management module receives uploaded logs through a log upload interface, and the log upload interface provides a log upload page and supports manual upload mode and scheduled upload mode.

[0022] The log management module stores the parsed logs in the database according to date classification; the log management module stores the parsed logs in the database according to device type classification; the log management module stores the parsed logs in the database according to log type classification.

[0023] The data interaction module uses a data query statement to set a query condition to filter the logs in the database, and the query condition supports filtering across devices, across a long time range, and based on different fields of multiple devices.

[0024] The log analysis module sets different parsing rules for logs of different devices, different types, and different versions. The parsing rules at least include: field type and type, storage requirements, and trigger period.

[0025] The log analysis module outputs the log analysis result through an analysis result output interface.

[0026] The present invention provides a management service for rail transit equipment logs, realizes the integration and unified management of rail transit equipment logs, facilitates the organization, summarization, archiving, backup and corresponding authority management of log information, and is conducive to log query and analysis. The entrances and responsibilities for uploading, analysis, and configuration are separated to facilitate the use of different personnel. Based on the Zeppelin service, it supports the use of formalized language to query, count and analyze logs. Based on big data analysis, it supports log query and statistics of multiple equipment types and multiple time spans, supports analysis of different business needs, improves the efficiency of data analysis, and expands the scope of data analysis. The output of analysis results is provided, which can be used by other systems or components to display or further utilize the log analysis results. Historical analysis results are provided to make the analysis results available for reuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of a rail transit equipment log correlation analysis platform provided by the present invention.

[0028] Figure 2 The present invention provides a flow chart of a method for correlation analysis of rail transit equipment logs.

[0029] Figure 3 This is a flowchart of log upload.

[0030] Figure 4 It is a flowchart of log query and interactive analysis.

[0031] Figure 5 It is a flowchart of log analysis based on parsing rules. DETAILED DESCRIPTION

[0032] The following is based on Figure 1 to Figure 5, specifically describe the preferred embodiments of the present invention.

[0033] Taking into account the shortcomings of traditional log analysis solutions, all log information is integrated into the same database. Based on the technical means of big data analysis and the concept of data governance, a unified log management and analysis service can be established to provide more efficient and multi-dimensional log analysis means and methods, improve log utilization and analysis efficiency, and achieve more efficient equipment protection and maintenance and more stable equipment operation.

[0034] like Figure 1 As shown, the present invention provides a rail transit equipment log correlation analysis platform, comprising:

[0035] The log management module 1 provides a log upload interface, and the log management module 1 is used to parse the logs uploaded through the log upload interface.

[0036] Database 2 is used to store parsed log information, which includes log type, date, and business fields. In this embodiment, a high-performance database is used, which is not limited to a relational database, a non-relational database, or an analytical database, but the query response time under high concurrency should be guaranteed to meet the user's requirements.

[0037] The data interaction module 3 provides a data processing engine interface. The data interaction module 3 accesses the data processing engine through the data processing engine interface, and queries, interactively analyzes and visualizes the log information in the database 2 through the data query statements of the data processing engine. In this embodiment, the data interaction module 3 uses the Zeppelin server. Zeppelin is an open source web-based notebook that provides interactive data analysis and visualization. The background supports access to multiple data processing engines, such as spark (an open source, fast, and general big data processing framework for distributed data processing and analysis), hive (a Hadoop-based data warehouse tool specifically used for processing and analyzing large-scale structured data), etc., and supports multiple languages: Scala (Apache Spark), Python (ApacheSpark), Spark SQL, Hive, Markdown, Shell, etc.

[0038] The log analysis module 4 provides a parsing rule configuration interface and an analysis result output interface. The log analysis module 4 configures different parsing rules through the parsing rule configuration interface, automatically analyzes the log information in the database 2 according to the parsing rules, and generates log analysis results. The log analysis results are output through the analysis result output interface to interact with other possible systems or components.

[0039] The log management module 1 and the log analysis module 4 support the use of multiple instances to expand their availability and ensure distributed consistency.

[0040] The association analysis platform supports user management, including upload permissions, query permissions, log management permissions, etc.

[0041] like Figure 2 As shown, the present invention also provides a method for correlation analysis of rail transit equipment logs, comprising:

[0042] Step S1, the log management module 1 receives the uploaded rail transit equipment log through the log upload interface of the log management module 1, the log management module 1 parses the uploaded log, automatically creates a table for the parsed log by date, by equipment type, and by log type, and stores it in the database 2.

[0043] The log upload interface provides a log upload page for providing log upload interface instructions and log upload options, and supports manual uploading through a page and scheduled uploading.

[0044] Step S2, the data interaction module 3 accesses the data processing engine through the data processing engine interface, writes data query statements through the data processing engine to query and interactively analyze the log information in the database 2, and visualizes the analysis results.

[0045] Among them, the query conditions are set using data query statements to filter the log information in the database 2, and the query conditions support filtering across devices, across time ranges, and based on different fields of multiple devices.

[0046] The data interaction module 3 uses a Zeppelin server, which provides a Zeppelin page and can filter and display data using corresponding data query statements based on the connected data processing engine.

[0047] Step S3, the log analysis module 4 configures different analysis rules through the analysis rule configuration interface, automatically analyzes the log information in the database 2 according to the analysis rules, generates log analysis results, and outputs the log analysis results through the analysis result output interface.

[0048] Set different parsing rules for logs of different devices, types, and versions, automatically analyze the logs according to the parsing rules, and generate log analysis results.

[0049] The parsing rule configuration interface provides a parsing rule setting page, which can set parsing rules for a certain device, a certain type of log, or a certain version, such as field type and category, storage requirements, etc.

[0050] like Figure 3 As shown, in one embodiment of the present invention, step S1 specifically includes the following steps:

[0051] Step S1.1, the log uploader logs in to the log upload page of the log management module using the account and password to control the project upload authority;

[0052] Step S1.2, select the corresponding project, the corresponding equipment type, the corresponding log type, and enter the corresponding log upload interface, for example, select SH project and select ZC equipment;

[0053] Step S1.3, upload log file A, log file B and log file C required by the ZC device of the SH project;

[0054] Step S1.4, the log management module parses log file A, log file B, and log file C. If the parsing fails, proceed to step S1.5; if the parsing succeeds, proceed to step S1.6;

[0055] Step S1.5, the log management module returns a parsing failure result and instructs the log uploader to re-upload the log and proceed to step S1.2;

[0056] Step S1.6: The log management module returns a successful analysis result and completes the log upload.

[0057] like Figure 4 As shown, in one embodiment of the present invention, step S2 specifically includes the following steps:

[0058] Step S2.1, the log analyst logs in to the Zeppelin page of the data interaction module using the account and password to control the project log data query permissions;

[0059] Step S2.2, select the corresponding project, such as SH project;

[0060] Step S2.3, the data interaction module configures the data source. For example, if the MySQL data source is used, the JDBC connection address (Connection URL, which is a string used to specify how to connect to a specific database), MySQL database account and password, and JDBC connector corresponding to MySQL are configured;

[0061] Step S2.4, the data interaction module creates a corresponding notebook (note), and configures the data source created in step S2.3;

[0062] Step S2.5, write a formal language in the data interaction module to query, for example, use the SQL language to query the time points when a Boolean variable is 0 in multiple ZC devices within the last 30 days;

[0063] Step S2.6: Use the query results of step S2.5 to perform further analysis to complete data query and analysis.

[0064] like Figure 5 As shown, in one embodiment of the present invention, step S3 specifically includes the following steps:

[0065] Step S3.1, the log parsing rule configuration personnel log in to the parsing rule setting page of the log analysis module using the account and password to manage the project configuration permissions;

[0066] Step S3.2, select the corresponding project, such as SH project;

[0067] Step S3.3, configure parsing rules, including specific formal statements, trigger cycles, etc. For example, for 10 ZC devices, query the number of times and time points when a variable exceeds the specified range in the past 7 days, query once a day and count;

[0068] Step S3.4, enable the parsing rules configured in step S3.3;

[0069] Step S3.5: The log analysis module completes the analysis of the log according to the parsing rules, and outputs the log analysis result through the analysis result output interface.

[0070] The present invention provides a management service for rail transit equipment logs, realizes the integration and unified management of rail transit equipment logs, facilitates the organization, summarization, archiving, backup and corresponding authority management of log information, and is conducive to log query and analysis. The entrances and responsibilities for uploading, analysis, and configuration are separated to facilitate the use of different personnel. Based on the Zeppelin service, it supports the use of formalized language to query, count and analyze logs. Based on big data analysis, it supports log query and statistics of multiple equipment types and multiple time spans, supports analysis of different business needs, improves the efficiency of data analysis, and expands the scope of data analysis. The output of analysis results is provided, which can be used by other systems or components to display or further utilize the log analysis results. Historical analysis results are provided to make the analysis results available for reuse.

[0071] It should be noted that, in the embodiments of the present invention, the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0072] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0073] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0074] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0075] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0076] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0077] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as limiting the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention. After reading the above content, it will be obvious for those skilled in the art to make various modifications and substitutions to the present invention. Therefore, the scope of protection of the present invention should be defined by the attached claims.

Claims

1. A rail transit equipment log correlation analysis platform, characterized in that: Include: Log management module, used to parse uploaded logs; Database, used to store parsed logs; A data interaction module provides a data processing engine interface. The data interaction module accesses the data processing engine through the data processing engine interface and queries, interactively analyzes and visualizes the logs in the database through the data query statements of the data processing engine. The log analysis module provides a parsing rule configuration interface. The log analysis module configures different parsing rules through the parsing rule configuration interface, automatically analyzes the logs in the database according to the parsing rules, and generates log analysis results.

2. The rail transit equipment log correlation analysis platform according to claim 1, characterized in that: The log management module provides a log upload interface, and the log upload interface is used to upload logs to the log management module.

3. The rail transit equipment log correlation analysis platform according to claim 1, characterized in that: The database at least includes: a relational database, a non-relational database, and an analytical database.

4. The rail transit equipment log correlation analysis platform according to claim 1, characterized in that: The data interaction module adopts Zeppelin server.

5. The rail transit equipment log correlation analysis platform according to claim 1, characterized in that: The data processing engine at least includes: spark, hive, and the languages ​​supported by the data processing engine at least include: Scala, Python, SparkSQL, Hive, Markdown, and Shell.

6. The rail transit equipment log correlation analysis platform according to claim 1, characterized in that: The log analysis module provides an analysis result output interface, through which the log analysis result is output.

7. A method for correlation analysis of rail transit equipment logs implemented based on the correlation analysis platform for rail transit equipment logs as described in any one of claims 1 to 6, characterized in that: Include: The log management module parses the uploaded logs and stores the parsed logs in the database; The data interaction module accesses the data processing engine through the data processing engine interface, writes data query statements through the data processing engine to query and interactively analyze the logs in the database, and visualizes the analysis results; The log analysis module configures different analysis rules through the analysis rule configuration interface, automatically analyzes the logs in the database according to the analysis rules, and generates log analysis results.

8. The method for correlation analysis of rail transit equipment logs according to claim 7, characterized in that: The log management module receives uploaded logs through a log upload interface, and the log upload interface provides a log upload page and supports manual upload mode and scheduled upload mode.

9. The method for correlation analysis of rail transit equipment logs according to claim 7, characterized in that: The log management module stores the parsed logs in the database according to date classification; the log management module stores the parsed logs in the database according to device type classification; the log management module stores the parsed logs in the database according to log type classification.

10. The method for correlation analysis of rail transit equipment logs according to claim 7, characterized in that: The data interaction module uses a data query statement to set a query condition to filter the logs in the database, and the query condition supports filtering across devices, across a long time range, and based on different fields of multiple devices.

11. The method for correlation analysis of rail transit equipment logs according to claim 7, characterized in that: The log analysis module sets different parsing rules for logs of different devices, different types, and different versions. The parsing rules at least include: field type and type, storage requirements, and trigger period.

12. The method for correlation analysis of rail transit equipment logs according to claim 7, characterized in that: The log analysis module outputs the log analysis result through an analysis result output interface.

Images