Read-write separated log recording and analyzing method

Abstract

The invention discloses a read-write separated log recording and analyzing method, and relates to the technical field of log data read-write, and the method comprises the following steps: responding to the receiving of log data, obtaining a spliced byte array according to the log data in a preset time sequence based on a preset splicing strategy, storing the spliced byte array in a preset database, and storing the spliced byte array in the preset database; updating paging information of a preset database; and in response to receiving of the query statement, converting the query statement into an abstract syntax tree based on a statement analysis algorithm, performing reverse query on paging information in a preset database based on the abstract syntax tree, and outputting log data meeting the abstract syntax tree. The method has the beneficial effects that the binary log data is directly written into the file, the log writing operation and the log analysis operation are separated, the log analysis operation is put in the query reading stage, and the log data recording efficiency is improved based on the fact that the operation frequency of querying the data by the user is far smaller than the frequency of writing the log data into the file.

Description

Technical Field

[0001] This application relates to the technical field of log data reading and writing, and particularly to a log recording and parsing method with read-write separation. Background Art

[0002] In the implementation of the urban rail signal system, the normal operation of the on-vehicle controller system (CC) and the zone controller (ZC) is crucial. Currently, R & D personnel mainly monitor the operation status and locate problems of the above systems through logs. However, the current log recording tool is implemented based on the following general log recording design scheme, and its performance and efficiency cannot be greatly improved.

[0003] In related technologies, the recording and parsing of log data usually adopt the following steps: 1. The tool collects the log data of the real CC and real ZC through the serial port and Ethernet interface, and collects the log data of the simulated CC and simulated ZC through the UDP interface. The log data includes the header of multiple fields with log characteristics, a log text information content encoded in ASCII, and a string of binary message data; 2. Parse each log data, and store the header log feature fields, log information content, and binary message data into a structure or object. The header feature fields include log ID, source, timestamp, log level, module, etc.; 3. Concatenate the attributes of each structure or object into an SQL statement (Insert statement); 4. Execute the above-generated SQL statement through the Sqlite database engine, write the data into the database, and finally save it to the hard disk.

[0004] However, the processing methods of related technologies face significant performance bottlenecks and resource waste problems, such as the following aspects: 1. Resource waste. It is almost impossible for on-site operation personnel and R & D and testing personnel to pay attention to all log data at a certain moment. Only when conducting problem location analysis, they need to observe the logs near the time period when the problem occurs. The vast majority of log data is not used, but needs to be recorded. In the above technical solution, each log is parsed once, concatenated into an SQL once, and an SQL insertion operation is performed once. Log parsing, SQL concatenation, and database insertion are all CPU-intensive operations, resulting in huge CPU resource occupancy during the operation of the tool, which will affect the stable operation of other software. During the testing process of the signal system, business software such as simulated CC and simulated ZC runs on the same computer as the log tool. Excessive resource occupancy by the log tool may affect the stable operation of these software; 2. Log loss. The urban rail transit signal system generates tens of thousands of log data per second. The speed of log reception will be much greater than the speed of writing logs to the disk. Before writing the generated logs per second, multiple operations such as parsing, SQL splicing, and database insertion are required, all of which are CPU-intensive operations. Although the reception and writing operations of log data can be completed separately in two threads, the slow writing speed will affect the reception efficiency and success rate. Since the network interface of the logs does not use a stable and reliable network protocol (due to efficiency reasons), when the tool load is too high and the reception is not timely, the packet loss rate will increase.

[0005] In summary, the algorithms used in existing log recording and parsing tools have serious technical bottlenecks when dealing with massive log data and are difficult to meet the requirements of efficiency and reliability. Summary of the Invention

[0006] In view of the problem in the prior art that it is impossible to balance the efficiency and reliability of log recording and parsing, the present application provides a log recording and parsing method with read-write separation. By splicing the received log data into a spliced byte array and directly writing the binary log data into a file, the log writing operation and the log parsing operation are separated, and the log parsing operation is placed in the query reading stage. Based on the fact that the operation frequency of user query data is much less than the frequency of writing log data into the file, the efficiency of log data recording is improved. At the same time, the written binary data file is more compact than the Sqlite database file, has a lower packet loss rate, and stronger reliability.

[0007] To achieve the above technical objectives, a technical solution provided by the present application is a log recording and parsing method with read-write separation, including the following steps: in response to the reception of log data, obtain a spliced byte array based on the log data within a preset time sequence according to a preset splicing strategy, store the spliced byte array in a preset database, and update the paging information of the preset database; in response to the reception of a query statement, convert the query statement into an abstract syntax tree based on a statement analysis algorithm, perform a reverse query on the paging information in the preset database based on the abstract syntax tree, and output the log data that meets the abstract syntax tree.

[0008] Further, the step of obtaining a spliced byte array based on the log data within a preset time sequence according to a preset splicing strategy in response to the reception of log data includes: in response to the reception of log data, if the preset time sequence has not been reached, write the log data into a cache list; if the preset time sequence has been reached, splice all the log data in the cache list based on the preset splicing strategy to obtain a spliced byte array, and clear the cache list.

[0009] Further, the preset splicing strategy includes: for each piece of log data, calculate the array length of the log data, serialize the array length into a byte array corresponding to the array length based on big-endian order; splice the byte array corresponding to the array length and the log data to obtain a spliced byte array.

[0010] Further, the serializing the array length into a byte array corresponding to the array length further includes: serializing the array length into a byte array of a preset number of bytes based on big-endian order.

[0011] Further, storing the spliced byte array into a preset database and updating the paging information of the preset database includes: storing the spliced byte array into the corresponding page of the preset database according to a preset time sequence, and updating the header paging information of the preset database.

[0012] Further, it further includes: in response to the writing or reading of the same paging data, locking the thread by using a read-write lock mechanism.

[0013] Further, in response to receiving a query statement, converting the query statement into an abstract syntax tree based on a statement analysis algorithm includes: in response to receiving a query statement, streamlining the query statement based on a preset deletion strategy to obtain a query expression; converting the query expression into an abstract syntax tree based on a statement analysis algorithm.

[0014] Further, performing a reverse query on the paging information in the preset database based on the abstract syntax tree and outputting the log data that meets the abstract syntax tree includes: obtaining the paging information in the preset database, and obtaining the spliced byte array in the preset database in reverse order; parsing the spliced byte array to obtain the corresponding log data; judging whether the parsed log data meets the query condition according to the abstract syntax tree, and if so, outputting the parsed log data.

[0015] Further, judging whether the parsed log data meets the query condition according to the abstract syntax tree includes: if the parsed log data matches the abstract syntax tree, adding the parsed log data to the query cache; if the number of log data in the query cache meets the query quantity, it is considered that the query condition is met.

[0016] Further, performing a reverse query on the paging information in the preset database based on the abstract syntax tree and outputting the log data that meets the abstract syntax tree further includes: if the number of log data in the query cache does not meet the query quantity, judging whether the spliced byte array of the current page has been completely read, and if so, obtaining the spliced byte array of the next page, and if not, continuing to read the next spliced byte array.

[0017] Advantages of the present application: By concatenating the received log data into a concatenated byte array and directly writing the binary log data into a file, the log writing operation and the log parsing operation are separated, improving the log writing efficiency. At the same time, the occupied storage space is reduced, and the storage efficiency is improved. Meanwhile, reverse query is performed using the paging information of the preset database, and the latest log data that the user is more concerned about is preferentially parsed. By reading in pages, the amount of data read at one time is reduced, the memory occupancy is reduced, and the reading efficiency is improved. Since the operation frequency of the user querying data is much lower than the frequency of writing log data into the file, by placing the parsing process of the log data in the query reading stage, a large amount of CPU resources are saved, and the overall load of the system is reduced. Brief Description of the Drawings

[0018] Figure 1 is a flowchart showing a method for log recording and parsing with read-write separation according to the present application Figure 1 。

[0019] Figure 2 is a flowchart showing a method for log recording and parsing with read-write separation according to the present application Figure 2 。 Detailed Embodiments

[0020] To make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only the best embodiments of the present application, which are only used to explain the present application and do not limit the protection scope of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0021] As Figure 1 、 Figure 2 shown, a method for log recording and parsing with read-write separation includes the following steps: In response to the reception of log data, obtain a concatenated byte array based on the log data within a preset time sequence according to a preset concatenation strategy, store the concatenated byte array in a preset database, and update the paging information of the preset database; In response to the reception of a query statement, convert the query statement into an abstract syntax tree based on a statement analysis algorithm, perform a reverse query on the paging information in the preset database based on the abstract syntax tree, and output the log data that meets the abstract syntax tree.

[0022] In this embodiment, by splicing the received log data into a spliced byte array and directly writing the binary log data into a file, the log writing operation and the log parsing operation are separated, improving the log writing efficiency. At the same time, the storage space occupation is reduced and the storage efficiency is improved. Meanwhile, the paging information of the preset database is used for reverse query, and the latest log data that the user is more concerned about is preferentially parsed. By reading in pages, the amount of data read at one time is reduced, the memory occupation is reduced, and the reading efficiency is improved. Since the operation frequency of the user querying data is much lower than the frequency of the log data being written into the file, by placing the parsing process of the log data in the query reading stage, a large amount of CPU resources are saved and the overall load of the system is reduced.

[0023] Specifically, in response to the reception of log data, obtaining a spliced byte array based on the log data within a preset time sequence according to a preset splicing strategy includes: In response to the reception of log data, if the preset time sequence has not been reached, the log data is written into the cache list. If the preset time sequence has been reached, based on the preset splicing strategy, all the log data in the cache list is spliced to obtain a spliced byte array, and the cache list is cleared.

[0024] After receiving the log data through UDP, network interface, serial port, etc., the log data is written into the cache list. When the preset time sequence is reached, all the log data in the cache list is spliced into a larger byte array, and the cache list is cleared. The cache list manages all the log data within the preset time sequence to ensure the orderliness of the log data and at the same time ensure the flexibility of log processing. The preset time sequence can be set according to the actual software storage and load capacity.

[0025] Among them, the preset splicing strategy includes: For each piece of log data, calculate the array length of the log data, and serialize the array length into a byte array corresponding to the array length; Based on the byte array corresponding to the array length and the log data, a spliced byte array is obtained.

[0026] For each piece of log data, calculate its length. If the log data is a string, in Java, the `length()` method can be used to obtain the number of characters. If the log data is a byte array, the `length` property can be used to obtain the number of bytes. Serialize the array length of the log data into a byte array, and concatenate the byte array corresponding to the array length with the log data. In this embodiment, place the serialized length byte array in front of the corresponding log data to obtain a concatenated byte array such as "the length of log data 1, log data 1, the length of log data 2, log data 2...". Thus, during subsequent query processes, it is possible to first read the byte array corresponding to the array length to determine the length of the next log, and according to the query conditions, determine whether the length of the next log meets the query conditions, so that log data with a length that does not meet the query conditions can be directly skipped without further parsing of the log data, improving the log parsing efficiency, reducing the use of CPU and memory, simplifying the query logic, and enhancing flexibility.

[0027] In some embodiments, serializing the array length into a byte array corresponding to the array length further includes: Serializing the array length into a byte array of a preset number of bytes based on big-endian order.

[0028] Big-endian (also known as big byte order) means that the high-order bytes of data are stored at low addresses, and the low-order bytes are stored at high addresses. Serialize the array length of the log data into a byte array of a preset number of bytes, and place the most significant byte of the length at the front. During the parsing process, the length field can be parsed sequentially from the beginning, simplifying the parsing logic and facilitating quick reading and judgment of the length of the log data.

[0029] In this embodiment, the preset number of bytes is two bytes. Since the maximum value that two bytes can represent is \(2^{16}-1\), that is, 65535, this is sufficient for the lengths of most log data.

[0030] In some other embodiments, the preset number of bytes can also be variable-length bytes, that is, one or more bytes are used to represent the length according to the actual length of the log data, and the high bit of the first byte is used to indicate whether there are subsequent bytes.

[0031] In this embodiment, the preset database at least includes a header describing paging information and several page contents. The paging information at least includes a page number, a starting position, and a length. Storing the concatenated byte array into the preset database and updating the paging information of the preset database includes: Storing the concatenated byte array into the corresponding page of the preset database according to a preset time sequence, and updating the header paging information of the preset database.

[0032] In this embodiment, the preset time sequence is 1 second. At this time, all the log data in the cache list is concatenated into a concatenated byte array every second, and each page stores the concatenated byte array of each second. A database file can write multiple target byte arrays, and the header paging information is updated every time one is written.

[0033] Among them, a read-write separation log recording and parsing method further includes: In response to the writing or reading of the same paging data, a thread is locked using a read-write lock mechanism.

[0034] When a thread needs to read a certain paging data, it will try to acquire a read lock. If no other thread holds the write lock for this paging data, then this thread can successfully acquire the read lock and start reading the data. When a thread needs to write a certain paging data, it will try to acquire a write lock. Only when no other thread holds the read lock or write lock for this paging data can this thread successfully acquire the write lock and start writing the data. Different paging data can be read or written by different threads simultaneously, but only one thread is supported to read or write the same paging data. Thus, multiple threads are allowed to read the same paging data simultaneously, improving the concurrency performance of the system. At the same time, by restricting the concurrent writing of the same paging data, data corruption or loss is prevented, and the consistency and integrity of the data are maintained.

[0035] In response to the reception of a query statement, converting the query statement into an abstract syntax tree based on a statement analysis algorithm includes: in response to the reception of a query statement, streamlining the query statement based on a preset streamlining strategy to obtain a query expression; Converting the query expression into an abstract syntax tree based on a statement analysis algorithm.

[0036] In this embodiment, since the log information is in a single table and there is no need for cross-table queries or addition, deletion, modification operations, the query statement is specifically streamlined through a preset streamlining strategy. The preset streamlining strategy can be set according to actual requirements. For example, based on the log data of a single table, the table name is omitted, unnecessary parts are removed, and only the core filtering conditions are retained. For example, if the SQL query statement is SELECT * FROM Message WHERE Module = 37 AND Type = 0, the query expression is streamlined to Module = 37 AND Type = 0. Thus, in the subsequent parsing process, the parsing of complex query statements is converted into the parsing of query expressions, reducing the parsing difficulty and improving the parsing efficiency.

[0037] When the user calls the input query statement through the UI or the automation interface, the query statement is streamlined according to the preset deletion strategy to obtain a query expression. At this time, the statement analysis algorithm includes a lexical analysis algorithm and a syntax analysis algorithm. Converting the query expression into an abstract syntax tree based on the statement analysis algorithm includes: Decompose the query expression into lexical units based on the lexical analysis algorithm; Organize the lexical units into an abstract syntax tree based on the syntax analysis algorithm.

[0038] The abstract syntax tree is a tree-like representation of the abstract syntax structure of the source code. It shows the syntax structure of the source code in the form of a tree, and each node represents a syntax element in the source code, such as expressions, statements, functions, etc. By constructing the abstract syntax tree, the source code is converted into a form that is easier to process and analyze, facilitating subsequent parsing and processing.

[0039] Perform a reverse query on the paging information in the preset database based on the abstract syntax tree, and output the log data that meets the abstract syntax tree, including: Obtain the paging information in the preset database, and obtain the concatenated byte array in the preset database in reverse order; Parse the concatenated byte array to obtain the corresponding log data; Judge whether the parsed log data meets the query conditions according to the abstract syntax tree. If so, output the parsed log data.

[0040] During the query process, read the header information of the preset database file to obtain paging information, and retrieve the page content from the preset database in reverse order according to the paging information. The page content is the pre-stored concatenated byte array. When no new page content can be obtained, return the query cache to the user. Each page content contains multiple log data, and there is a length field before each log data. Parse the log data according to the length field, and sort the parsed log data in reverse order of time to ensure that the user sees the latest log records first. According to the matching situation between the abstract syntax tree and the parsed log data, select the log data that matches the abstract syntax tree for output to complete the query.

[0041] Judging whether the parsed log data meets the query conditions according to the abstract syntax tree includes: If the parsed log data matches the abstract syntax tree, add the parsed log data to the query cache; If the number of log data in the query cache meets the query quantity, it is considered that the query conditions are met.

[0042] Obtain a log data, parse the header feature fields, log information content, and binary message data of the log data, and store them in a structure or object. Use the previous abstract syntax tree to determine whether the structure or object meets the query conditions. If it meets the conditions, add the structure or object to the query cache. If the number of structures or objects in the query cache meets the query quantity, return the query cache to the user and clear the query cache.

[0043] Among them, the reverse query of the paging information in the preset database based on the abstract syntax tree, and the output of the log data that meets the abstract syntax tree also includes: If the number of log data in the query cache does not meet the query quantity, determine whether the concatenated byte array of the current page has been completely read. If so, obtain the concatenated byte array of the next page. If not, continue to read the next concatenated byte array.

[0044] In this embodiment, during the process of logging, there is no need to perform operations such as parsing, SQL concatenation, Sqlite engine parsing SQL and inserting data, etc. After collecting the log data, it is directly written to a file, avoiding a large number of unnecessary operations and greatly reducing the CPU occupancy rate.

[0045] At the same time, after collecting log data according to a certain time or quantity, the tool directly writes the log to a file without going through a complex SQL concatenation process. Since the directly written data is binary data, it is more compact than the data written after being processed by the Sqlite database engine, and can reduce the disk occupancy by more than 50% in practical applications, thus greatly reducing the CPU occupancy and disk occupancy. And because the writing is more efficient, the reception of log data is also more stable and reliable, and the probability of packet loss is reduced.

[0046] Since a traditional relational database (such as Sqlite) is not used to store data in this embodiment, users cannot use SQL statements to query log data. Therefore, a preset deletion strategy is constructed based on the query requirements, so as to facilitate the filtering of query statements and obtain a more concise query expression for subsequent parsing. When the user queries log data using the query expression, the present invention will first parse the query expression into an abstract syntax tree, and then read the log data from the preset database paging, filtering while parsing using the abstract syntax tree until enough log data that meets the conditions is obtained and returned to the user. The parsing process of log data is completely placed in the query reading stage, and the operation frequency of the user querying data is much less than the frequency of writing log data to a file, thus saving a lot of CPU resources.

[0047] The above - described specific implementation manners are the preferred implementation manners of a log recording and parsing method with read - write separation in this application. It does not limit the specific scope of this application in this way. The scope of this application includes but is not limited to this specific implementation manner. All equivalent changes made according to the shape and structure of this application are within the protection scope of this application.

Claims

1. A log recording and parsing method for read-write separation, characterized in that: It includes the following steps: In response to the reception of log data, obtain a spliced byte array based on the log data within a preset time sequence according to a preset splicing strategy, store the spliced byte array in a preset database, and update the paging information of the preset database; In response to the reception of a query statement, convert the query statement into an abstract syntax tree based on a statement analysis algorithm, perform a reverse query on the paging information in the preset database based on the abstract syntax tree, and output the log data that meets the abstract syntax tree.

2. A read-write separated log recording and parsing method as claimed in claim 1, wherein: The step of obtaining a spliced byte array based on the log data within a preset time sequence according to a preset splicing strategy in response to the reception of log data includes: In response to the reception of log data, if the preset time sequence has not been reached, write the log data into a cache list. If the preset time sequence has been reached, splice all the log data in the cache list according to the preset splicing strategy to obtain a spliced byte array, and clear the cache list.

3. A read-write separated log recording and parsing method as claimed in claim 1, wherein: The preset splicing strategy includes: For each piece of log data, calculate the array length of the log data, and serialize the array length into a byte array corresponding to the array length based on big-endian order; Splice the byte array corresponding to the array length and the log data to obtain a spliced byte array.

4. A read-write separated log recording and parsing method as claimed in claim 3, wherein: The step of serializing the array length into a byte array corresponding to the array length further includes: Serializing the array length into a byte array of a preset number of bytes based on big-endian order.

5. A read-write separated log recording and parsing method as claimed in claim 1, wherein: The step of storing the spliced byte array in a preset database and updating the paging information of the preset database includes: Store the spliced byte array into the corresponding page of the preset database according to the preset time sequence, and update the header paging information of the preset database.

6. The log recording and parsing method with read-write separation according to claim 1, wherein: It further includes: In response to the writing or reading of the same paging data, use a read-write lock mechanism to lock the thread.

7. A read-write separated log recording and parsing method as claimed in claim 1, wherein: The step of converting the query statement into an abstract syntax tree based on a statement analysis algorithm in response to the reception of the query statement includes: In response to the reception of the query statement, streamline the query statement based on a preset deletion strategy to obtain a query expression; Convert the query expression into an abstract syntax tree based on the statement analysis algorithm.

8. A read-write separated log recording and parsing method as claimed in claim 1, wherein: The step of performing a reverse query on the paging information in the preset database based on the abstract syntax tree and outputting the log data that meets the abstract syntax tree includes: Obtain the paging information in the preset database, and obtain the spliced byte array in the preset database in reverse order; Parse the spliced byte array to obtain the corresponding log data; Judge whether the parsed log data meets the query condition according to the abstract syntax tree. If so, output the parsed log data.

9. A log recording and parsing method with read-write separation as described in claim 8, characterized in that: The step of determining whether the parsed log data meets the query conditions according to the abstract syntax tree includes: If the parsed log data matches the abstract syntax tree, add the parsed log data to the query cache; If the number of log data in the query cache meets the query quantity, it is considered that the query conditions are met.

10. A log recording and parsing method with read-write separation as described in claim 8, characterized in that: The step of performing a reverse query on the paging information in the preset database based on the abstract syntax tree and outputting the log data that meets the abstract syntax tree further includes: If the number of log data in the query cache does not meet the query quantity, determine whether the concatenated byte array of the current page has been completely read. If so, obtain the concatenated byte array of the next page. If not, continue to read the next concatenated byte array.

Images