A method and device for parsing LCU system log data based on python and mysql

Abstract

The application provides a method and device for parsing LCU system log data based on python and mysql, which comprises the following steps: importing a message type template, adding the successfully imported message type template to a mysql database to form a template table; importing a log file into the mysql database, splitting the log file into multiple messages, and storing field data into an interactive display list based on a python script; selecting a message to be parsed in the interactive display list, matching the selected message with a message type model in the template table; parsing each field data in the message based on the matched message type template, and processing the parsing result into a visual list through the python script. The application only needs to import the log into the system, and then the test personnel can select the hexadecimal data to be parsed according to the needs, so that the parsed data can be obtained, the test efficiency of the test personnel is greatly improved, and the test personnel can conveniently analyze the data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data analysis, and particularly relates to a method and device for analyzing LCU system log data based on python and mysql. BACKGROUND

[0002] Currently, some metro trains in the rail transit industry use on-board logic control units (LCU) to realize train electrical control, which has the characteristics of convenient programming and flexible control mode. The LCU system mostly uses binary and hexadecimal data transmission methods when communicating, and the log messages generated during its operation are mostly stored in hexadecimal data. Hexadecimal storage method can not only represent a variety of device states and communication parameter values using limited numerical symbols, but also can be flexibly converted into binary and octal data, while also reducing the memory space occupied by data storage and improving the running efficiency.

[0003] However, hexadecimal data is difficult to read, and requires high professional skills. Non-professionals cannot easily understand the data, which affects work efficiency and increases work difficulty. When testing personnel and equipment maintenance personnel check the device running data and device state, they need to parse the log content according to the corresponding data type and data interface protocol document bit by bit, and each new log file needs to be re-parsed to extract the latest data value. If there are too many log file data, it will take a lot of time to find the relevant data. SUMMARY

[0004] In order to solve the shortcomings and deficiencies of hexadecimal data in the log message of the LCU system being obscure, low parsing efficiency, and long manual time consumption in the prior art, the present application proposes a method for parsing LCU system log data based on python and mysql, which is used to parse the hexadecimal data of the log message in the LCU system of the metro train, mainly realized by python and mysql technology. MySQL is used to store imported log templates and automatically generate log temporary tables to store current imported log data when importing logs. Through multi-table query matching of the log temporary table and the template table, the message type, field name, field value and other data can be correctly displayed in the foreground for user screening and analysis. Python is mainly used to provide user interface to display data and human-computer interaction, etc. Users can perform data operations such as importing and querying according to the interface, and realize related operations such as jumping between different interfaces, connecting mysql, data processing, data display, internal related logic control of pages, etc.

[0005] The method for parsing LCU system log data based on python and mysql proposed by the present application includes:

[0006] Importing the message type template, adding the successfully imported message type template to the mysql database to form a template table;

[0007] Importing the log file into the mysql database, splitting the log file into multiple messages, and storing the field data into an interactive display list based on a python script;

[0008] Selecting the message to be parsed in the interactive display list, matching the selected message with the message type model in the template table, parsing each field data in the message based on the matched message type template, and processing the parsing result into a visual list through a python script.

[0009] Optionally, the message type template is an excel template table, and the excel template table defines a template id, a message type, a field name, a data length, a field position of a plurality of abnormal data, and a corresponding field meaning based on an interface protocol of an LCU system.

[0010] Optionally, the matching of the selected message with the message type model in the template table comprises: determining the message type according to a message header of the message, and matching the message type template corresponding to the same message type with the message.

[0011] Optionally, the parsing of each field data in the message based on the matched message type template and the processing of the parsing result into a visual list through a python script comprise:

[0012] Constructing a table header of the visual list according to the field meaning defined by the message type template in the order of the field position;

[0013] According to the data length defined by the message type template, splitting the message into a plurality of field data, and adding the split field data to the table header of the corresponding field position to obtain the visual list.

[0014] Optionally, when the parsing result is processed into a visual list through a python script, it is judged whether the parsed field data is abnormal data defined in the message type template, and if it is abnormal data, the background color of the corresponding position in the visual list is marked yellow.

[0015] Optionally, the method further comprises setting a high-level search interface based on a python script, and the high-level search interface comprises a search result display list and a plurality of search engines corresponding to different search conditions.

[0016] Optionally, after the matched message type template is used to parse each field data in the message, the parsing result generated each time is stored into the myspl database to form a historical parsing database, and the search engine is used to search the historical parsing database to obtain the parsing result meeting the search condition and display the search result in a search result display list.

[0017] The application also provides a device for parsing LCU system log data based on python and mysql, which comprises:

[0018] The template importing unit is used for importing message type templates and adding the successfully imported message type templates to the mysql database to form a template table.

[0019] The log importing unit is used for importing log files into the mysql database, splitting the log files into multiple messages, and storing the field data into an interactive display list based on a python script.

[0020] The matching unit is used for selecting the message to be parsed in the interactive display list and matching the selected message with the message type templates in the template table.

[0021] The parsing unit is used for parsing each field data in the message based on the matched message type template and processing the parsing result into a visual list through a python script.

[0022] The application provides the technical scheme with the beneficial effects as follows.

[0023] (1) According to the LCU system log data parsing method, the tester does not need to extract the message according to the message type in the log, but only needs to import the log into the system, and then selects the hexadecimal data to be parsed according to the needs, so that the parsed data can be obtained, the testing efficiency of the tester is greatly improved, the tester does not need to be familiar with the relevant data interface protocol document, and the LCU system log can be quickly parsed, the meaning of each data in the hexadecimal data can be displayed according to the imported message type template, and the tester can conveniently analyze the data.

[0024] (2) For the abnormal value in the data, the system can automatically identify and mark the bottom color of the data bit as yellow, and the data analysis efficiency is improved.

[0025] (3) The data can be flexibly displayed according to the message period, type and other fields, and the tester can conveniently check the real-time change process of the data.

[0026] (4) Different data types only need to be imported into the template, and the compatibility and portability are high. DETAILED DESCRIPTION

[0027] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the following embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0028] Figure 1 A flowchart of a method for parsing LCU system log data based on python and mysql is provided for the present embodiment.

[0029] Figure 2 A main interface diagram of the log parsing platform is provided for the present embodiment.

[0030] Figure 3 A high-level search interface of the log parsing platform is provided for the present embodiment.

[0031] Figure 4 An architecture diagram of a device for parsing LCU system log data based on python and mysql is provided for the present embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.

[0033] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0035] It should be understood that, in the present application, "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units need not be limited to those clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be understood that, in the present application, "a plurality of" means two or more. "And / or" is only a description of the association between the associated objects, which means that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "Including A, B and C" and "including A, B, C" means that A, B and C are all included, "including A, B or C" means that one of A, B and C is included, and "including A, B and / or C" means that any one or any two or three of A, B and C is included.

[0037] It should be understood that, in the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B" or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but also can be determined according to A and / or other information. The matching of A and B is that the similarity of A and B is greater than or equal to a preset threshold.

[0038] Depending on the context, "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting".

[0039] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in some examples.

[0040] Example one:

[0041] As shown in the figure, the embodiment proposes a method for parsing LCU system log data based on python and mysql, which includes: S1: importing a message type template, adding the successfully imported message type template to the mysql database to form a template table; Figure 1

[0042] S2: importing the log file into the mysql database, splitting the log file into multiple messages, and storing the field data into the interactive display list based on the python script;

[0043] ​S3: Select the message that needs to be parsed in the interactive display list, match the selected message with the message type model in the template table;

[0044] S4: Based on the matched message type template, parse each field data in the message, and process the parsing result into a visual list through a python script.

[0045] In the LCU system, currently, the test personnel needs to obtain the LCU.db format log file, and needs to manually find the required message type, which increases the difficulty of work, and the disadvantages are as follows: there are many different types of messages in the LCU log every period, and it is impossible to accurately view the change of a single data in different periods; the data bits of different periods in the LCU log are various, and it is impossible to accurately identify the meaning of each data bit; if you want to find a certain byte when the data is long, you can only parse from the first bit according to the LCU system data interface protocol document, which is time-consuming and laborious; the newly added test personnel needs to be familiar with all the data interface protocol documents first, and needs to understand the difficult protocol before parsing the data, which consumes time and requires high requirements.

[0046] In view of the above-mentioned disadvantages, the embodiment is mainly applicable to the hexadecimal message data generated in the operation log of the subway train logic control system, the log is imported into the system, and the hexadecimal message to be parsed is selected, so as to realize the rapid parsing of the hexadecimal message, thereby solving the parsing difficulty caused by the change of the log file in the LCU system with different periods.

[0047] The embodiment is based on a log parsing platform opened by python and mysql technology, the main interface of the log parsing platform is as shown in Figure 2 The interactive buttons set on the main interface include existing message type query, import template and import log, before using the method proposed in the embodiment to parse the LCU system log, the required data template can be manually imported into the log parsing platform in advance, so that the log parsing platform is automatically applicable to different data formats. After the test personnel obtains the LCU system log.db file, the log file is imported into the log parsing platform, the log parsing platform automatically generates a temporary table for saving all data contents of the file, and all field data in the file is displayed in the form of a list in the foreground page of the log parsing platform. The test personnel selects the hexadecimal data that needs to be parsed, and after clicking, the log parsing platform will automatically parse the selected data and jump to the data parsing page, which details the value of each field data bit for the test personnel to analyze the data. The system provides an advanced search function, which can search according to the period number and the data type, improves the work efficiency of the test personnel, and improves the usability of the system.

[0048] In the embodiment, the existing message type query button is clicked, and the content shown in Table 1 is displayed in the interface.

[0049] Table 1

[0050] Message Type Type Name 0x83 Service Synchronization Message In 0x84 External Connection State 0x85 External Connection Message In 0x87 Extensible Platform State 0x8B External Connection Network State

[0051] The tester can obtain the message type template imported in the log analysis platform through the message type query, and then import the unimported message type template into the log analysis platform according to the need by clicking to upload the file. The prepared message type template is selected, and then the "import template" button is clicked to import the file into the mysql database, and the log analysis platform will automatically pop up the "import success" or "format error" prompt box.

[0052] In the embodiment, the message type template is an excel template table, and the excel template table defines the field position and corresponding field meaning of the template id, message type, field name, data length and a plurality of abnormal data based on the interface protocol of the LCU system.

[0053] Based on the above definition of the message type template, the selected message is matched with the message type model in the template table in the embodiment, including: determining the message type according to the message header of the message, and matching the message type template corresponding to the same message type with the message.

[0054] The message type template is matched with the message type template, and the message type template is matched with the message type template.

[0055] The field meaning defined by the message type template is constructed into the table header of the visual list according to the order of the field position.

[0056] According to the data length defined by the message type template, the message is parsed and split into a plurality of field data, and the split field data is added to the table header of the corresponding field position to obtain the visual list.

[0057] Taking the data type in the LCU system as the business software state message and the type field as DATAID_APPSTAT (0xC2) as an example, the data validity of the message is 12. The data interface protocol corresponding to the message is shown in Table 2.

[0058] Table 2

[0059]

[0060] The excel template table generated based on the above protocol is shown in Table 3.

[0061] Table 3

[0062]

[0063] In one example, the log parsing platform obtains the hexadecimal message "c280 0c 00 00 00 00 0b 55 55 00 00 39 24 71eb" from the LCU runtime log .db file. Referring to the message type template shown in Table 3, the message is split into the fields "c2", "80", "0c 00", "00 00 00 0b", "55", "55", "0000", and "39 24 71eb" according to the data length definition, and then mapped to the fields defined in the template to obtain the parsing results shown in Table 4 below.

[0064] Table 4

[0065]

[0066]

[0067] In this embodiment, when the parsing results are processed into a visual list using a Python script, it is determined whether the parsed field data is abnormal data defined in the message type template. If it is abnormal data, the background color is highlighted in yellow at the corresponding position in the visual list to improve data analysis efficiency.

[0068] In this embodiment, to facilitate testers in checking the real-time changes in data and to enable flexible display of log data based on fields such as message period and type, the method further includes setting up an advanced search interface based on a Python script. This advanced search interface includes a search result display list and multiple search engines corresponding to different search criteria. Specifically, after parsing the data in each field of the message based on the matched message type template, the generated parsing results are stored in the myspl database to form a historical parsing database. The search engines then retrieve parsing results that meet the search criteria from the historical parsing database and display them in the search result display list.

[0069] In this embodiment, the advanced search interface is as follows: Figure 3 As shown, the advanced search function offers two search engines: one based on message type and the other on periodicity. At least one search engine must be selected for advanced searching. The message type search engine allows filtering by selecting the desired message type from a drop-down menu, while the periodicity search engine allows filtering by entering a specific periodicity range. The results will then be displayed in a list format, showing the parsed results that match the search criteria. It's understandable that when both search engines are selected and both message type and periodicity are set as search criteria, the displayed results will only include those matching both the specified message type and periodicity range.

[0070] In the embodiment, different message type import is flexible, the use range is controllable, the hexadecimal log data of the LCU system can be automatically parsed according to the existing message template type, the hexadecimal data which is harsh and difficult to read is divided according to the field, and then displayed in the form of list, clear and easy to understand, improve the readability, and the abnormal data in the log can be automatically identified according to the abnormal value in the template, and the data is marked yellow, improve the data analysis efficiency.

[0071] The embodiment can also set different search schemes according to the use scene, meet the diversified needs of users.

[0072] The embodiment is based on the current mainstream python language, has the advantages of simple maintenance and wide application range, can be applied to different operating systems such as windows and linux, and has strong usability and portability.

[0073] Embodiment two:

[0074] As shown in Figure 4 The embodiment proposes a device 5 for parsing LCU system log data based on python and mysql, which comprises: a template import unit 51 for importing message type templates, adding the successfully imported message type templates to the mysql database to form a template table;

[0075] A log import unit 52 is used for importing log files into the mysql database, splitting the log files into multiple messages, and storing the field data into an interactive display list based on a python script;

[0076] A matching unit 53 is used for selecting the message to be parsed in the interactive display list, and matching the selected message with the message type model in the template table;

[0077] An analysis unit 54 is used for parsing each field data in the message based on the matched message type template, and processing the parsing result into a visual list through a python script.

[0078] In the LCU system, currently, a tester needs to obtain a log file in the LCU.db format, and needs to manually find the required message type, thereby increasing the difficulty of work, and the disadvantages are specifically as follows: there are multiple different types of messages in the LCU log per cycle, and the change of a single data in different cycles cannot be accurately viewed; the data bits of different cycles in the LCU log are various, and the meaning of each data bit cannot be accurately identified; when the data is long, if a certain byte is to be found, the LCU system data interface protocol document needs to be parsed from the first bit, which is time-consuming and laborious; a newly added tester needs to be familiar with all the data interface protocol documents, and needs to understand the difficult protocol to parse the data, which is time-consuming and requires high requirements.

[0079] In view of the above disadvantages, the embodiment is mainly applicable to hexadecimal message data generated in the operation log of a subway train logic control system, the log is imported into the system, the hexadecimal message to be parsed is selected, the hexadecimal message is quickly parsed, and the difficulty of parsing caused by the change of the log file in the LCU system with different cycles is solved.

[0080] The log parsing platform based on the python and mysql technologies is opened, a main interface of the log parsing platform is as shown in Figure 2 The interactive buttons arranged on the main interface include an existing message type query, an import template, and an import log, before the LCU system log is parsed by using the method, the required data template can be manually imported into the log parsing platform in advance, so that the log parsing platform is automatically applicable to different data formats. After the tester obtains the LCU system log.db file, the log file is imported into the log parsing platform, the log parsing platform automatically generates a temporary table for saving all data contents of the file, and all field data in the file is displayed in a list manner in the foreground page of the log parsing platform. The tester selects the hexadecimal data to be parsed, and after the hexadecimal data is clicked, the log parsing platform automatically parses the selected data and jumps to a data parsing page, the data parsing page details the value of the data bit in each field, and the data is provided for the tester to analyze the data. The system provides an advanced search function, the cycle number and the data type can be searched, the work efficiency of the tester is improved, and the system usability is improved.

[0081] In the embodiment, the message type template is an excel template table, the excel template table is based on the interface protocol of the LCU system, and defines the field position and corresponding field meaning of the template id, the message type, the field name, the data length, and a plurality of abnormal data.

[0082] Based on the definition of the message type template, the selected message is matched with the message type model in the template table by the matching unit 53, including: determining the message type according to the message header of the message, and matching the message type template corresponding to the same message type with the message.

[0083] The analysis unit 54 is specifically used for:

[0084] The field meaning defined by the message type template is constructed into the table header of the visual list according to the order of the field position.

[0085] According to the data length defined by the message type template, the message is split into multiple field data, and the split field data is added to the table header of the corresponding field position to obtain the visual list.

[0086] In the embodiment, when the analysis unit 54 processes the analysis result into a visual list through a python script, it is judged whether the field data parsed is abnormal data defined in the message type template, and if it is abnormal data, the bottom color of the corresponding position in the visual list is marked yellow.

[0087] In the embodiment, in order to facilitate the test personnel to check the real-time change process of the data, realize the function of flexible display of log data according to the message period, type and other fields, the device further includes a high-level search interface based on a python script, and the high-level search interface includes a search result display list and a plurality of search engines corresponding to different search conditions. After the field data in the message is analyzed based on the matched message type template, the analysis result generated each time is stored in the myspl database to form a historical analysis database, and the search engine retrieves the analysis result meeting the search condition in the historical analysis database and displays it in the search result display list.

[0088] In the embodiment, the high-level search interface is as shown in Figure 3 Two search engines are set in the high-level search, which are the search engines with the message type as the search condition and the period number as the search condition. At least one of the search engines is selected when the high-level search is performed. The search engine with the message type as the search condition can select the message type to be searched through a drop-down menu to realize screening, and the search engine with the period number as the search condition can input the period number to limit the range to realize screening. Finally, the analysis result meeting the search condition is displayed in the form of a list in the interface. It can be understood that when the two search engines are selected at the same time to set the search conditions of the message type and the period number, the analysis result displayed by the search is the analysis result meeting the set message type and period number range at the same time.

[0089] The various serial numbers in the above embodiments are merely for description, and do not represent the assembly or use sequence of the components.

[0090] The above description is only examples of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for parsing LCU system log data based on python and mysql, characterized in that, The method includes: Import message type templates and add successfully imported message type templates to the MySQL database to form a template table. The message type template is an Excel template table. The Excel template table is based on the interface protocol of the LCU system and defines the field positions and corresponding field meanings, including template ID, message type, field name, data length, and several abnormal data fields. Import the log file into the MySQL database, split the log file into multiple messages, and store the field data into an interactive display list using a Python script; Select the message to be parsed from the interactive display list, and match the selected message with the message type model in the template table; Based on the matched message type template, the data of each field in the message is parsed, and the parsing results are processed into a visual list using a Python script; The process involves parsing the data in each field of the message based on the matched message type template, and then using a Python script to process the parsing results into a visual list, including: The meanings of the fields defined in the message type template are arranged in the order of their positions to form the header of a visual list; Based on the data length defined in the message type template, the message is parsed and split into multiple fields. The split field data is then added to the header of the corresponding field to obtain a visual list that displays the value of each data bit in each field for testers to analyze the data.

2. The method for parsing LCU system log data based on python and mysql according to claim 1, characterized in that, The step of matching the selected message with the message type model in the template table includes: determining the message type based on the message header, and matching the message type template with the same message type with the message. 3.The method of parsing LCU system log data based on python and mysql according to claim 2, wherein, When processing the parsed results into a visual list using a Python script, it is determined whether the parsed field data is abnormal data defined in the message type template. If it is abnormal data, the corresponding position in the visual list is highlighted in yellow.

4. The method for parsing LCU system log data based on python and mysql according to claim 1, characterized in that, The method also includes setting up an advanced search interface based on a Python script. The advanced search interface includes a list of search results and multiple search engines corresponding to different search criteria.

5. The method for parsing LCU system log data based on python and mysql according to claim 4, characterized in that, After parsing the data of each field in the message based on the matched message type template, the parsing results generated each time are stored in the myspl database to form a historical parsing database. The search engine retrieves the parsing results that meet the search conditions from the historical parsing database and displays them in the search results display list.

6. A device for parsing LCU system log data based on python and mysql, characterized in that, The device includes: Template Import Unit: Used to import message type templates, and adds successfully imported message type templates to the MySQL database to form a template table; Log Import Unit: Used to import log files into a MySQL database, split the log files into multiple messages, and store the field data in an interactive display list based on a Python script. The message type template is an Excel template table, which is based on the interface protocol of the LCU system and defines the template ID, message type, field name, data length, and the field positions and corresponding meanings of several abnormal data. The matching unit is used for selecting a message needing to be parsed in the interactive display list, and matching the selected message with a message type model in a template table; The parsing unit is used for parsing each field data in the message based on the matched message type template, and processing the parsing result into a visual list through a python script; The parsing unit is also used for constructing a table header of the visual list according to the sequence of field positions of the field meanings defined by the message type template, splitting the message into multiple field data according to the data length defined by the message type template, adding the split field data to the table header of the corresponding field position, obtaining the visual list, and displaying the value of the data bit in each field data, so as to analyze the data by the tester.

Images