Log data query method and system, terminal and computer readable storage medium

Abstract

The invention discloses a log data query method and system and a terminal, and the method comprises the steps: obtaining log information, and filtering the log information to obtain log data; storing the log data in a preset storage space according to a preset storage rule; and in response to a log query request initiated by a user, querying the plurality of log data in the storage space according to the log query request to obtain a query result. By obtaining the log information, filtering the log information, storing the log information in the preset space, responding to the query request, retrieving the log data in the storage space and returning the query result to the user, the efficient query of the embedded system is realized, the log data query efficiency is greatly improved, and quick and accurate log information is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log data query method, system, terminal and computer-readable storage medium. Background Art

[0002] Existing embedded systems are usually equipped with small storage space. Traditional log management methods may lead to storage resource exhaustion. The processor performance of embedded systems is limited. Traditional log management methods may occupy too many computing resources, affecting the real-time performance of the system. Embedded systems are usually used in sensitive scenarios, such as security equipment, medical equipment, industrial control systems, etc. The security of log data is crucial. The log query method is single and the query efficiency is low. This leads to low query efficiency of embedded systems, resulting in low efficiency of querying log data, thereby affecting the efficiency of log data query and analysis.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] The main purpose of the present invention is to provide a log data query method, system and computer-readable storage medium, aiming to solve the problem of low query efficiency of embedded system in the prior art, which leads to low efficiency of querying log data.

[0005] To achieve the above object, the present invention provides a log data query method, which includes the following steps:

[0006] Obtaining log information, filtering the log information, and obtaining log data;

[0007] Storing the log data in a preset storage space according to a preset storage rule;

[0008] In response to a log query request initiated by a user, a plurality of log data in the storage space are queried according to the log query request to obtain a query result.

[0009] Optionally, the log data query method, wherein the acquiring log information and filtering the log information to obtain the log data, specifically includes:

[0010] Collect log information from the log system and obtain filtering rules in the preset log collection configuration file;

[0011] Invalid log information in the log information is filtered according to the filtering rules in the preset log collection configuration file to obtain log data of the log information.

[0012] Optionally, the log data query method, wherein storing the log data in a preset storage space according to a preset storage rule, specifically includes:

[0013] According to the log data matching compression algorithm, the log data is compressed by the compression algorithm to obtain target data;

[0014] According to the log level corresponding to the target data, the target data is stored in different partitions of a preset storage space.

[0015] Optionally, the log data query method, wherein storing the target data in different partitions of a preset storage space according to the log level corresponding to the target data, specifically includes:

[0016] Obtaining the log level corresponding to the target data, and distinguishing the target data according to the priority of the log level corresponding to the target data to obtain high priority data and low priority data;

[0017] Based on the queue-jumping priority storage strategy, the high-priority data is stored in a first partition of a preset storage space, and the low-priority data is stored in a second partition of a preset storage space;

[0018] The log levels include: debug level, information level, warning level, error level and assertion level.

[0019] Optionally, the log data query method, wherein the responding to the log query request initiated by the user, querying the plurality of log data in the storage space according to the log query request to obtain the query result, specifically includes:

[0020] Obtaining the timestamp, event type and keyword of the log query request sent by the user;

[0021] A matching query is performed on the plurality of log data in the storage space according to the timestamp, the event type and the keyword to obtain a query result.

[0022] Optionally, the log data query method further comprises:

[0023] When the log data is stored in a preset storage space, the log data is encrypted according to a preset symmetric encryption key to obtain encrypted data.

[0024] Optionally, the log data query method further comprises:

[0025] Divide the storage space into regions to obtain the number of partitions, and determine whether the number of partitions reaches the target number of partitions;

[0026] If the number of partitions reaches the target number of partitions, the log data of the first partition of the storage space is overwritten.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also provides a log data query system, wherein the log data query system:

[0028] A log data acquisition module is used to acquire log information, filter the log information, and obtain log data;

[0029] A log data storage module, used to store the log data in a preset storage space according to a preset storage rule;

[0030] The log data query module is used to respond to a log query request initiated by a user, query the multiple log data in the storage space according to the log query request, and obtain a query result.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a log data query program, and when the log data query program is executed by a processor, the steps of the log data query method described above are implemented.

[0032] In the present invention, log information is obtained, the log information is filtered and processed to obtain log data; the log data is stored in a preset storage space according to a preset storage rule; a log query request initiated by a user is responded to, and multiple log data in the storage space are queried according to the log query request to obtain a query result. The present invention obtains log information, filters it, stores it in a preset space, responds to a query request, retrieves the log data in the storage space, and returns the query result to the user, thereby realizing efficient querying of the embedded system, greatly improving the efficiency of querying log data, and providing users with fast and accurate log information. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a preferred embodiment of the log data query method of the present invention;

[0034] Figure 2 It is a flow chart of a preferred embodiment of the log data query system of the present invention;

[0035] Figure 3 It is a schematic diagram of the principle of a preferred embodiment of the log data query system of the present invention;

[0036] Figure 4It is a schematic diagram of the composition of the terminal control device in a preferred embodiment of the log data query system of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] In existing embedded systems, log records are essential for monitoring system operation status, diagnosing faults, and ensuring safety. Embedded systems usually have limited resources, including storage space, processing efficiency, and real-time performance. As a result, embedded systems are usually equipped with small storage space. Traditional log management methods may cause storage resources to be exhausted, thus affecting the query efficiency of log data. Therefore, a method for efficient query efficiency and fast query of log data in embedded systems is needed.

[0039] The log data query method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the log data query method includes the following steps:

[0040] Step S10: Obtain log information, filter the log information, and obtain log data.

[0041] Specifically, log information of the log system is collected, and filtering rules in a preset log collection configuration file are obtained; invalid log information in the log information is filtered according to the filtering rules in the preset log collection configuration file to obtain log data of the log information (by setting the filtering rules in the preset log collection configuration file, only log information of key events of the system is collected to reduce the number of unnecessary logs to adapt to a small storage space).

[0042] Step S20: storing the log data in a preset storage space according to a preset storage rule.

[0043] Specifically, according to the log data matching compression algorithm, the log data is compressed by the compression algorithm (the compression algorithm adopts QuickLZ to reduce the space occupied by the original log to save storage space) to obtain the target data; according to the log level corresponding to the target data, the target data is stored in different partitions of the preset storage space; the log level corresponding to the target data is obtained, and the target data is distinguished according to the priority of the log level corresponding to the target data to obtain high priority data and low priority data (the log data is divided into high priority and low priority for storage, and the high priority log is queued and stored first); based on the queue-jumping priority storage strategy, the high priority data is stored in the first partition of the preset storage space, and the high priority data is stored in the first partition of the preset storage space. The low priority data is stored in the second partition of the preset storage space; the log level includes: debugging level, information level, warning level, error level and assertion level (the preset output log level includes debugging, information, warning, error, assertion and other log levels, the later the level, the higher the priority, the earlier the storage), further, the log data storage process adopts a queue method, and the log data is written to the memory when the system is idle, which does not affect the real-time performance of the key tasks of the system; when the system crashes, the log data is immediately written to the memory (when the system outputs log data, the log data is now saved to RAM, RAM is very fast, consumes very little time, but will be lost in power failure, and the log data will be written to the memory when the system is idle, and the memory will not be lost in power failure).

[0044] Step S30: respond to the log query request initiated by the user, query the multiple log data in the storage space according to the log query request, and obtain a query result.

[0045] Specifically, the timestamp, event type and keyword of the log query request sent by the user are obtained; a matching query is performed on the multiple log data in the storage space according to the timestamp, the event type and the keyword to obtain a query result (supporting quick query of log data by conditions such as timestamp, event type and keyword).

[0046] Furthermore, when the log data is stored in a preset storage space, the log data is encrypted according to a preset symmetric encryption key to obtain encrypted data (symmetric encryption technology is used when storing logs to ensure the security of log content, AES128 encryption is adopted, and the encrypted data is compressed after encryption to obtain the encrypted data. When decrypting, it is necessary to decompress it first and then decrypt it to obtain the original log data. The encryption and decryption keys are written during production testing. Each device has a separate key, and only the corresponding key can be decrypted. For example: when the system is idle, the log is encrypted by the AES128 encryption algorithm to encrypt the original log data, and then compressed by QuickLZ The log is compressed by the compression algorithm, and is first decompressed and then decrypted during decryption); the storage space is divided into regions to obtain the number of partitions, and it is determined whether the number of partitions reaches the target number of partitions; if the number of partitions reaches the target number of partitions, the log data of the first partition of the storage space is overwritten (the log data is stored in a circular overwriting storage manner, and the first stored log data is automatically overwritten to keep the latest log to adapt to the limited storage space. For example, the storage space is divided into five parts, the first partition is used first, and the fifth partition is used last. After use, the first partition will be reused to overwrite the log data of the first partition).

[0047] As an example, Figure 2 As shown, the embedded device outputs a log, determines whether it meets the storage conditions through the log level and filtering rules, and then caches the log to the queue. According to the log level, the high priority log is inserted into the head of the queue and stored first. When the idle task is running, it starts to take out data from the head of the queue, encrypts and compresses the data, and then writes it to different storage areas according to the log level. For particularly important notifications, an alarm is sent to the visualization device to notify the operation and maintenance personnel to handle it in time (real-time analysis of the collected log data, and notification of the system alarm log to the operation and maintenance personnel by transmitting it to the visualization device); when the log is read, different storage areas are indexed according to the different log levels read, and the log can be queried by timestamp range, event type and keyword. When an external device is inserted, the log data will be automatically backed up to the external storage device, which needs to be decompressed and decrypted with a key to obtain the plaintext log (supporting automatic backup of log data by the system when an external storage device is connected, reducing the risk of data loss when the memory is damaged).

[0048] Beneficial effects of the present invention:

[0049] (1) The present invention reduces the waste of storage space and ensures data security and storage space utilization by filtering, classifying, compressing and encrypting log data.

[0050] (2) The present invention adopts a queue and idle scheduling algorithm to store logs when the system is idle, thereby reducing the impact on the real-time performance of the system.

[0051] (3) The present invention improves the log query speed by storing different log levels in different partitions, and supports fast query of log data by conditions such as timestamp, event type and keyword.

[0052] Furthermore, if Figure 3 As shown, based on the above log data query method, the present invention also provides a log data query system, wherein the log data query system includes:

[0053] The log data acquisition module 51 is used to acquire log information and filter the log information to obtain log data;

[0054] A log data storage module 52 is used to store the log data in a preset storage space according to a preset storage rule;

[0055] The log data query module 53 is used to respond to a log query request initiated by a user, query the multiple log data in the storage space according to the log query request, and obtain a query result.

[0056] Furthermore, if Figure 4 As shown, based on the above-mentioned log data query method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0057] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a log data query program 40 is stored on the memory 20, and the log data query program 40 can be executed by the processor 10, thereby realizing the log data query method in the present application.

[0058] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the log data query method.

[0059] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0060] In one embodiment, when the processor 10 executes the log data query program 40 in the memory 20, the following steps are implemented:

[0061] Obtaining log information, filtering the log information, and obtaining log data;

[0062] Storing the log data in a preset storage space according to a preset storage rule;

[0063] In response to a log query request initiated by a user, a plurality of log data in the storage space are queried according to the log query request to obtain a query result.

[0064] The obtaining of log information and filtering of the log information to obtain log data specifically includes:

[0065] Collect log information from the log system and obtain filtering rules in the preset log collection configuration file;

[0066] Invalid log information in the log information is filtered according to the filtering rules in the preset log collection configuration file to obtain log data of the log information.

[0067] The storing of the log data in a preset storage space according to a preset storage rule specifically includes:

[0068] According to the log data matching compression algorithm, the log data is compressed by the compression algorithm to obtain target data;

[0069] According to the log level corresponding to the target data, the target data is stored in different partitions of a preset storage space.

[0070] The step of storing the target data in different partitions of a preset storage space according to the log level corresponding to the target data specifically includes:

[0071] Obtaining the log level corresponding to the target data, and distinguishing the target data according to the priority of the log level corresponding to the target data to obtain high priority data and low priority data;

[0072] Based on the queue-jumping priority storage strategy, the high-priority data is stored in a first partition of a preset storage space, and the low-priority data is stored in a second partition of a preset storage space;

[0073] The log levels include: debug level, information level, warning level, error level and assertion level.

[0074] The responding to the log query request initiated by the user, querying the plurality of log data in the storage space according to the log query request, and obtaining the query result specifically includes:

[0075] Obtaining the timestamp, event type and keyword of the log query request sent by the user;

[0076] A matching query is performed on the plurality of log data in the storage space according to the timestamp, the event type and the keyword to obtain a query result.

[0077] Wherein, the log data query method further includes:

[0078] When the log data is stored in a preset storage space, the log data is encrypted according to a preset symmetric encryption key to obtain encrypted data.

[0079] Wherein, the log data query method further includes:

[0080] Divide the storage space into regions to obtain the number of partitions, and determine whether the number of partitions reaches the target number of partitions;

[0081] If the number of partitions reaches the target number of partitions, the log data of the first partition of the storage space is overwritten.

[0082] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a log data query program, and when the log data query program is executed by a processor, the steps of the log data query method described above are implemented.

[0083] In summary, the present invention provides a log data query method, system and terminal, the method comprising: obtaining log information, filtering the log information to obtain log data; storing the log data in a preset storage space according to a preset storage rule; responding to a log query request initiated by a user, querying a plurality of log data in the storage space according to the log query request, and obtaining a query result. The present invention obtains log information, performs filtering processing, stores it in a preset space, responds to a query request, retrieves the log data in the storage space, and returns the query result to the user, thereby realizing efficient querying of an embedded system, greatly improving the efficiency of querying log data, and providing users with fast and accurate log information.

[0084] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or terminal system including the element.

[0085] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a terminal, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0086] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A log data query method, characterized in that: The log data query method comprises: Obtaining log information, filtering the log information, and obtaining log data; Storing the log data in a preset storage space according to a preset storage rule; In response to a log query request initiated by a user, a plurality of log data in the storage space are queried according to the log query request to obtain a query result.

2. The log data query method according to claim 1, characterized in that: The obtaining of log information and filtering of the log information to obtain log data specifically includes: Collect log information from the log system and obtain filtering rules in the preset log collection configuration file; Invalid log information in the log information is filtered according to the filtering rules in the preset log collection configuration file to obtain log data of the log information.

3. The log data query method according to claim 1, characterized in that: The storing the log data in a preset storage space according to a preset storage rule specifically includes: According to the log data matching compression algorithm, the log data is compressed by the compression algorithm to obtain target data; According to the log level corresponding to the target data, the target data is stored in different partitions of a preset storage space.

4. The log data query method according to claim 3, characterized in that: According to the log level corresponding to the target data, storing the target data in different partitions of a preset storage space specifically includes: Obtaining the log level corresponding to the target data, and distinguishing the target data according to the priority of the log level corresponding to the target data to obtain high priority data and low priority data; Based on the queue-jumping priority storage strategy, the high-priority data is stored in a first partition of a preset storage space, and the low-priority data is stored in a second partition of a preset storage space; The log levels include: debug level, information level, warning level, error level and assertion level.

5. The log data query method according to claim 1, characterized in that: The responding to the log query request initiated by the user, querying the plurality of log data in the storage space according to the log query request, and obtaining the query result specifically includes: Obtaining the timestamp, event type and keyword of the log query request sent by the user; A matching query is performed on the plurality of log data in the storage space according to the timestamp, the event type and the keyword to obtain a query result.

6. The log data query method according to claim 1, characterized in that: The log data query method also includes: When the log data is stored in a preset storage space, the log data is encrypted according to a preset symmetric encryption key to obtain encrypted data.

7. The log data query method according to claim 1, characterized in that: The log data query method also includes: Divide the storage space into regions to obtain the number of partitions, and determine whether the number of partitions reaches the target number of partitions; If the number of partitions reaches the target number of partitions, the log data of the first partition of the storage space is overwritten.

8. A log data query system, characterized in that: The log data query system comprises: A log data acquisition module is used to acquire log information, filter the log information, and obtain log data; A log data storage module, used to store the log data in a preset storage space according to a preset storage rule; The log data query module is used to respond to a log query request initiated by a user, query the multiple log data in the storage space according to the log query request, and obtain a query result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a log data query program stored in the memory and executable on the processor. When the log data query program is executed by the processor, the steps of the log data query method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a log data query program, and when the log data query program is executed by a processor, the steps of the log data query method according to any one of claims 1 to 7 are implemented.

Images