A read-write separated log recording and analyzing method

Abstract

The application discloses a log record and analysis method with read-write separation, and relates to the technical field of log data reading and writing, and comprises the following steps: in response to the reception of log data, a splicing byte array is acquired according to the log data in a preset time sequence based on a preset splicing strategy, the splicing byte array is stored in a preset database, and the paging information of the preset database is updated; in response to the reception of a query statement, the query statement is converted into an abstract syntax tree based on a statement analysis algorithm, the paging information in the preset database is queried in a reverse order based on the abstract syntax tree, and log data meeting the abstract syntax tree is output. The application has the beneficial effects that binary log data is directly written into a file, the log writing operation and the log analysis operation are separated, and the log analysis operation is placed in the query reading stage; the operation frequency of user query data is far less than the frequency of log data writing into the file, and the log data recording efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of log data reading and writing technology, and in particular to a log recording and parsing method with read-write separation. Background Technology

[0002] In the implementation of urban rail signaling systems, the normal operation of the onboard controller (CC) and zone controller (ZC) systems is crucial. Currently, developers primarily monitor the operational status and analyze problems in these systems using logs. However, current logging tools are based on a common logging design scheme, which limits their performance and efficiency.

[0003] The recording and parsing of log data in related technologies typically involves the following steps:

[0004] 1. The tool collects log data from the real device CC and ZC via serial port and Ethernet interface, and collects log data from the simulated CC and ZC via UDP interface. The log data includes a header with multiple fields representing log characteristics, a log text message content encoded in ASCII, and a string of binary message data;

[0005] 2. Parse each log data entry and store the header log feature fields, log information content, and binary message data into a structure or object. The header feature fields include log ID, source, timestamp, log level, module, etc.

[0006] 3. Concatenate each structure or object attribute into an SQL statement (Insert statement);

[0007] 4. Execute the generated SQL statements using the SQLite database engine to write data into the database and finally save it to the hard drive.

[0008] However, the processing methods of related technologies face significant performance bottlenecks and resource waste, such as in the following aspects:

[0009] 1. Resource waste: On-site operations personnel and R&D testing personnel are unlikely to monitor all log data at any given time. They only need to observe logs from the time period in which the problem occurred during problem localization and analysis. The vast majority of log data is unused but still needs to be recorded. In the above technical solution, each log entry is parsed once, SQL is concatenated once, and an SQL insert operation is performed once. Log parsing, SQL concatenation, and database insertion are all CPU-intensive operations, resulting in significant CPU resource consumption during tool operation, which can affect the stable operation of other software. During signal system testing, business software such as simulation CC and simulation ZC run on the same computer as the logging tool. Excessive resource consumption by the logging tool may affect the stable operation of these other software programs.

[0010] 2. Log loss: The urban rail signaling system generates tens of thousands of log data per second, and the log reception speed is much faster than the log writing speed to disk. Before writing each log, multiple operations are required, including parsing, SQL concatenation, and database insertion—all CPU-intensive. Although log data reception and writing can be completed in two separate threads, the slow writing speed affects reception efficiency and success rate. Furthermore, because the log network interface uses a non-stable and unreliable network protocol (for efficiency reasons), delayed reception under heavy load increases packet loss.

[0011] In summary, the algorithms used by existing log recording and parsing tools have serious technical bottlenecks when processing massive amounts of log data, making it difficult to meet the requirements of efficiency and reliability. Summary of the Invention

[0012] This application addresses the problem in existing technologies that cannot simultaneously achieve both efficient and reliable log recording and parsing. It provides a read-write separation log recording and parsing method. By concatenating received log data into a concatenated byte array and directly writing the binary log data to a file, the log writing and parsing operations are separated. The log parsing operation is placed in the query and reading stage. Since the frequency of user data query operations is much lower than the frequency of log data writing to the file, the efficiency of log data recording is improved. At the same time, the written binary data file is more compact than SQLite database files, with a lower packet loss rate and stronger reliability.

[0013] To achieve the above technical objectives, this application provides a technical solution: a read-write separation log recording and parsing method, comprising the following steps: in response to the receipt of log data, obtaining a concatenated byte array based on a preset concatenation strategy and log data within a preset time sequence, storing the concatenated byte array in a preset database, and updating the paging information of the preset database; in response to the receipt of a query statement, converting the query statement into an abstract syntax tree based on a statement analysis algorithm, performing a reverse query on the paging information in the preset database based on the abstract syntax tree, and outputting log data that satisfies the abstract syntax tree.

[0014] Furthermore, the step of obtaining a concatenated byte array based on the log data within a preset time sequence in response to the receipt of log data includes: in response to the receipt of log data, if the preset time sequence has not been reached, writing the log data into a cache list; if the preset time sequence has been reached, concatenating all log data in the cache list based on the preset concatenation strategy to obtain a concatenated byte array, and clearing the cache list.

[0015] Furthermore, the preset splicing strategy includes: for each log data, calculating the array length of the log data, serializing the array length into a byte array corresponding to the array length based on the large segment order; and splicing the byte array corresponding to the array length and the log data to obtain a spliced ​​byte array.

[0016] Furthermore, the step of serializing the array length into a byte array corresponding to the array length also includes: serializing the array length into a byte array of preset bytes based on big-endian order.

[0017] Furthermore, storing the concatenated byte array in a preset database and updating the paging information of the preset database includes: storing the concatenated byte array in the corresponding page of the preset database according to a preset timing sequence, and updating the header paging information of the preset database.

[0018] Furthermore, it also includes: locking threads using a read-write lock mechanism in response to writing or reading data from the same page.

[0019] Furthermore, the step of converting the query statement into an abstract syntax tree based on a statement analysis algorithm in response to receiving the query statement includes: simplifying the query statement based on a preset reduction strategy to obtain a query expression; and converting the query expression into an abstract syntax tree based on a statement analysis algorithm.

[0020] Furthermore, the step of performing a reverse query on the pagination information in the preset database based on the abstract syntax tree and outputting log data that satisfies the abstract syntax tree includes: obtaining the pagination information in the preset database and obtaining the concatenated byte array in the preset database in reverse order; parsing the concatenated byte array to obtain the corresponding log data; and determining whether the parsed log data satisfies the query conditions based on the abstract syntax tree. If so, the parsed log data is output.

[0021] Furthermore, the step of determining whether the parsed log data meets the query conditions based on the abstract syntax tree includes: if the parsed log data matches the abstract syntax tree, then the parsed log data is added to the query cache; if the number of log data in the query cache meets the query quantity, then the query conditions are considered to be met.

[0022] Furthermore, the step of performing a reverse query on the pagination information in the preset database based on the abstract syntax tree and outputting log data that satisfies the abstract syntax tree also includes: if the number of log data in the query cache does not meet the query quantity, then it is determined whether the concatenated byte array of the current page has been completely read. If yes, then the concatenated byte array of the next page is obtained; otherwise, the next concatenated byte array is read.

[0023] The beneficial effects of this application are as follows: By concatenating the received log data into a concatenated byte array and directly writing the binary log data to a file, the log writing and parsing operations are separated, improving log writing efficiency and reducing storage space usage, thus improving storage efficiency. Simultaneously, by utilizing the pre-defined pagination information of the database for reverse order querying, the latest log data that users are more concerned with is parsed first. Paginated reading reduces the amount of data read at one time, lowering memory usage and improving read efficiency. Since the frequency of user data query operations is much lower than the frequency of log data writing to the file, placing the log data parsing process in the query and read phase saves a significant amount of CPU resources and reduces the overall system load. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a read-write separation log recording and parsing method according to this application. Figure 1 .

[0025] Figure 2 This is a flowchart illustrating a read-write separation log recording and parsing method according to this application. Figure 2 . Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] like Figure 1 , Figure 2 As shown, a read-write separation log recording and parsing method includes the following steps:

[0028] In response to the receipt of log data, the system obtains a concatenated byte array based on the log data within a preset time sequence according to a preset concatenation strategy, stores the concatenated byte array in a preset database, and updates the paging information of the preset database.

[0029] In response to the received query statement, the query statement is converted into an abstract syntax tree based on the statement analysis algorithm. Based on the abstract syntax tree, the pagination information in the preset database is queried in reverse order, and log data that satisfies the abstract syntax tree is output.

[0030] In this embodiment, by concatenating the received log data into a concatenated byte array and directly writing the binary log data to a file, the log writing and parsing operations are separated, improving log writing efficiency and reducing storage space usage, thus improving storage efficiency. Simultaneously, reverse-order queries are performed using pre-defined database pagination information, prioritizing the parsing of the latest log data that users are more interested in. Paginated reading reduces the amount of data read at once, lowering memory usage and improving read efficiency. Since the frequency of user data queries is much lower than the frequency of log data writing to files, placing the log data parsing process in the query and read phase saves significant CPU resources and reduces the overall system load.

[0031] Specifically, in response to the receipt of log data, the concatenation byte array is obtained based on the log data within a preset time sequence according to a preset concatenation strategy, including:

[0032] In response to the receipt of log data, if the preset time sequence has not been reached, the log data is written to the cache list. If the preset time sequence has been reached, all log data in the cache list is concatenated according to the preset concatenation strategy to obtain the concatenated byte array, and the cache list is cleared.

[0033] After receiving log data via UDP, network interface, serial port, etc., the log data is written to a buffer list. When a preset time sequence is reached, all log data in the buffer list is concatenated into a larger byte array, and the buffer list is cleared. Managing all log data within the preset time sequence through the buffer list ensures the sequential nature of the log data while maintaining flexibility in log processing. The preset time sequence can be set according to the actual software storage and load capacity.

[0034] The preset splicing strategies include:

[0035] For each log data entry, calculate the array length of the log data and serialize the array length into a byte array corresponding to the array length.

[0036] The concatenated byte array is obtained by concatenating the byte array corresponding to the array length and the log data.

[0037] For each log entry, its length is calculated. If the log data is a string, the `length()` method can be used in Java to get the number of characters; if the log data is a byte array, the `length` property can be used to get the number of bytes. The array length of the log data is serialized into a byte array, and the byte array corresponding to the array length is concatenated with the log data. In this embodiment, the serialized length byte array is placed before the corresponding log data, resulting in a concatenated byte array such as "length of log data 1, log data 1, length of log data 2, log data 2...". This allows for subsequent queries to first read the byte array corresponding to the array length to determine the length of the next log entry. Based on the query conditions, it can be determined whether the length of the next log entry meets the query conditions, thus allowing log data whose length does not meet the query conditions to be skipped directly without further parsing. This improves log parsing efficiency, reduces CPU and memory usage, simplifies query logic, and enhances flexibility.

[0038] In some embodiments, serializing the array length into a byte array corresponding to the array length further includes:

[0039] The array length is serialized into a byte array of preset bytes based on big-endian order.

[0040] Big-Endian, also known as big-byte order, refers to the storage of the most significant byte at the lowest memory address and the least significant byte at the highest memory address. By serializing the length of the log data array into a byte array of a predetermined number of bytes and placing the most significant byte of the length first, the parsing process can start from the beginning and parse the length field sequentially, simplifying the parsing logic and facilitating rapid reading and determination of the log data length.

[0041] In this embodiment, the preset byte is two bytes. Since the maximum value that two bytes can represent is 2^16-1, or 65535, this is sufficient for the length of most log data.

[0042] In other embodiments, the preset byte can also be a variable-length byte, that is, one or more bytes are used to represent the length according to the actual log data length, and the high-order bits of the first byte are used to indicate whether there are any subsequent bytes.

[0043] In this embodiment, the preset database includes at least a header describing pagination information and several page contents. The pagination information includes at least a page number, a starting position, and a length. Storing the concatenated byte array in the preset database and updating the pagination information in the preset database includes:

[0044] The concatenated byte array is stored in the corresponding page of the preset database according to the preset timing, and the header page information of the preset database is updated.

[0045] In this embodiment, the preset timing is 1 second. At this time, all log data in the cache list is concatenated into a concatenated byte array every second, and each page stores the concatenated byte array for each second. A database file can write multiple target byte arrays, and the header page information is updated after each one is written.

[0046] One method for reading and writing separated log recording and parsing also includes:

[0047] In response to writing or reading data in the same page, a read-write lock mechanism is used to lock the thread.

[0048] When a thread needs to read data from a specific page, it attempts to acquire a read lock. If no other thread holds a write lock on that page, the thread can successfully acquire the read lock and begin reading the data. Similarly, when a thread needs to write data to a specific page, it attempts to acquire a write lock. Only if no other thread holds a read or write lock on that page can the thread successfully acquire the write lock and begin writing the data. Different pages can be read or written by different threads simultaneously, but the same page can only be read or written by one thread at a time. This allows multiple threads to read the same page simultaneously, improving system concurrency performance. Furthermore, by limiting concurrent writes to the same page, data corruption or loss is prevented, maintaining data consistency and integrity.

[0049] In response to the receipt of a query statement, the process of converting the query statement into an abstract syntax tree based on a statement analysis algorithm includes: in response to the receipt of a query statement, simplifying the query statement based on a preset reduction strategy to obtain a query expression;

[0050] The query expression is converted into an abstract syntax tree based on the statement analysis algorithm.

[0051] In this embodiment, since the log information is from a single table, there is no need for cross-table queries or CRUD operations. Therefore, a preset simplification strategy is used to specifically streamline the query statement. This preset simplification strategy can be set according to actual needs. For example, based on single-table log data, the table name can be omitted, unnecessary parts can be removed, and only the core filtering conditions can be retained. For example, the SQL query statement SELECT * FROM Message WHERE Module = 37 AND Type = 0 can be simplified to Module = 37 AND Type = 0. This transforms the parsing of complex query statements into the parsing of query expressions during subsequent parsing, reducing parsing difficulty and improving parsing efficiency.

[0052] When a user inputs a query statement via a UI or automated interface, the query statement is simplified according to a preset reduction strategy to obtain a query expression. At this point, the statement analysis algorithm includes lexical analysis and syntax analysis algorithms. Based on the statement analysis algorithm, the query expression is converted into an abstract syntax tree, including:

[0053] The query expression is decomposed into lexical units based on lexical analysis algorithms;

[0054] Lexical units are organized into an abstract syntax tree based on syntax analysis algorithms.

[0055] An abstract syntax tree (AST) is a tree-like representation of the abstract syntactic structure of source code. It displays the syntactic structure of the source code in a tree format, where each node represents a syntactic element in the source code, such as an expression, statement, or function. By constructing an AST, the source code is transformed into a more easily processed and analyzed form, thus facilitating subsequent parsing.

[0056] Based on the abstract syntax tree, a reverse query is performed on the pagination information in the preset database, and the output log data that satisfies the abstract syntax tree includes:

[0057] Retrieve pagination information from a preset database and retrieve the concatenated byte array from the preset database in reverse order;

[0058] Parse the concatenated byte array to obtain the corresponding log data;

[0059] The system uses an abstract syntax tree to determine whether the parsed log data meets the query conditions. If so, the parsed log data is output.

[0060] During the query process, the header information of a pre-defined database file is read to obtain pagination information. Based on this pagination information, page content is retrieved from the pre-defined database in reverse chronological order. The page content is a pre-stored concatenated byte array. If no new page content is found, the query cache is returned to the user. Each page contains multiple log entries, each preceded by a length field. The log data is parsed based on this length field and sorted in reverse chronological order to ensure the user sees the most recent log records first. Finally, based on the matching between the abstract syntax tree (AST) and the parsed log data, the log data matching the AST is selected and output, completing the query.

[0061] Determining whether the parsed log data meets the query conditions based on the abstract syntax tree includes:

[0062] If the parsed log data matches the abstract syntax tree, then the parsed log data is added to the query cache;

[0063] If the amount of log data in the query cache meets the query limit, then the query condition is considered met.

[0064] Retrieve a log entry, parse its header feature fields, log message content, and binary message data, and store them in a structure or object. Use the previously obtained abstract syntax tree to determine if the structure or object meets the query criteria. If it does, add the structure or object to the query cache. If the number of structures or objects in the query cache meets the query requirements, return the cached data to the user and clear the cache.

[0065] The process of performing a reverse query on the pagination information in the preset database based on the abstract syntax tree, and outputting log data that satisfies the abstract syntax tree, also includes:

[0066] If the number of log data in the query cache does not meet the query limit, check if the concatenated byte array of the current page has been completely read. If yes, obtain the concatenated byte array of the next page; otherwise, continue reading the next concatenated byte array.

[0067] In this embodiment, during the log recording process, there is no need to perform operations such as parsing, SQL concatenation, SQL engine parsing and data insertion. After collecting the log data, it is directly written to the file, avoiding a large number of unnecessary calculations and greatly reducing the CPU usage.

[0068] Furthermore, after collecting log data at certain intervals or in certain quantities, the tool directly writes the logs to a file without requiring complex SQL concatenation. Because the data written directly is binary data, it is more compact than data written after processing by the SQLite database engine, reducing disk usage by more than 50% in practical applications, thus significantly reducing CPU and disk usage. Moreover, due to the more efficient writing process, log data reception is more stable and reliable, reducing the probability of packet loss.

[0069] Since this embodiment does not use a traditional relational database (such as SQLite) to store data, users cannot query log data using SQL statements. Therefore, a preset deletion strategy is constructed based on query requirements to facilitate filtering of query statements, resulting in more concise query expressions that are easier to parse subsequently. When a user uses a query expression to query log data, this invention first parses the query expression into an abstract syntax tree, then reads log data from the preset database in pages, filtering using the abstract syntax tree as it parses, until enough log data meeting the conditions is obtained and returned to the user. The log data parsing process is entirely handled in the query and reading phase, while the frequency of user data queries is much lower than the frequency of log data being written to files, thus saving significant CPU resources.

[0070] The specific embodiments described above are preferred embodiments of a read-write separation log recording and parsing method of this application, and are not intended to limit the specific implementation scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of this application are within the protection scope of this application.

Claims

1. A read-write separation log recording and parsing method, characterized in that: The steps include the following: In response to the receipt of log data, the system obtains a concatenated byte array based on the log data within a preset time sequence according to a preset concatenation strategy, stores the concatenated byte array in a preset database, and updates the pagination information of the preset database. In response to the received query statement, the query statement is converted into an abstract syntax tree based on the statement analysis algorithm. Based on the abstract syntax tree, the pagination information in the preset database is queried in reverse order, and log data that satisfies the abstract syntax tree is output.

2. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: The step of responding to the receipt of log data and obtaining a concatenated byte array based on log data within a preset time sequence according to a preset concatenation strategy includes: In response to the receipt of log data, if the preset time sequence has not been reached, the log data is written to the cache list. If the preset time sequence has been reached, all log data in the cache list is concatenated according to the preset concatenation strategy to obtain the concatenated byte array, and the cache list is cleared.

3. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: The preset splicing strategy includes: For each log data entry, calculate the array length of the log data, and serialize the array length into a byte array corresponding to the array length based on big-endian order; The concatenated byte array is obtained by concatenating the byte array corresponding to the array length and the log data.

4. The log recording and parsing method with read-write separation as described in claim 3, characterized in that: The step of serializing the array length into a byte array corresponding to the array length also includes: The array length is serialized into a byte array of preset bytes based on big-endian order.

5. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: The step of storing the concatenated byte array into a preset database and updating the pagination information in the preset database includes: The concatenated byte array is stored in the corresponding page of the preset database according to the preset timing, and the header page information of the preset database is updated.

6. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: Also includes: In response to writing or reading data in the same page, a read-write lock mechanism is used to lock the thread.

7. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: The step of converting the query statement into an abstract syntax tree based on a statement analysis algorithm in response to the receipt of the query statement includes: In response to the received query statement, the query statement is simplified based on a preset reduction strategy to obtain the query expression; The query expression is converted into an abstract syntax tree based on the statement analysis algorithm.

8. The log recording and parsing method with read-write separation as described in claim 1, characterized in that: The step of performing a reverse query on the pagination information in the preset database based on the abstract syntax tree, and outputting log data that satisfies the abstract syntax tree, includes: Retrieve pagination information from a preset database and retrieve the concatenated byte array from the preset database in reverse order; Parse the concatenated byte array to obtain the corresponding log data; The system uses an abstract syntax tree to determine whether the parsed log data meets the query conditions. If so, the parsed log data is output.

9. The log recording and parsing method with read-write separation as described in claim 8, characterized in that: The step of determining whether the parsed log data meets the query conditions based on the abstract syntax tree includes: If the parsed log data matches the abstract syntax tree, then the parsed log data is added to the query cache; If the amount of log data in the query cache meets the query limit, then the query condition is considered met.

10. The log recording and parsing method with read-write separation as described in claim 8, characterized in that: The step of performing a reverse query on the pagination information in the preset database based on the abstract syntax tree, and outputting log data that satisfies the abstract syntax tree, also includes: If the number of log data in the query cache does not meet the query limit, check if the concatenated byte array of the current page has been completely read. If yes, obtain the concatenated byte array of the next page; otherwise, continue reading the next concatenated byte array.

Images