Log file storage and reading method and device

Abstract

The invention relates to a log file storage and reading method and device. The method comprises the following steps: acquiring daily operation data of a system to generate log data; combining the log data according to a preset format to generate a log file; storing the log file in a cache according to a preset directory structure; generating a log statistical information file, and storing an intraday memory statistical information file in the log statistical information file in a memory in real time; and when a preset condition is met, storing the log file in the cache into a hard disk so as to facilitate reading. According to the log file storage and reading method and device, an efficient log storage and reading mechanism suitable for an embedded system with limited resources is provided, hard disk writing is optimized through memory cache, rapid access of massive logs is achieved, hardware equipment is protected, and the real-time performance and reliability of data are ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of computer information processing, and in particular to a method and device for storing and reading log files. Background Art

[0002] In various websites, platforms and software systems, the storage of logs or data records usually relies on database systems. For medium and large systems, relational databases such as Oracle, MySQL, and PostgreSQL are often used. When the amount of data is huge, the system will use distributed databases or distributed storage methods to improve performance and scalability. However, in embedded systems, due to limited memory and CPU resources, lightweight databases such as SQLite are usually used. SQLite takes up less resources, is easy to integrate, and is suitable for processing small-scale data. However, when the amount of data increases (such as more than 1 million log records), the query efficiency of SQLite will drop significantly, and may even cause the system process to crash. Therefore, embedded systems have higher requirements for the storage and fast query of massive logs.

[0003] In large and medium-sized platforms, databases such as MySQL, PostgreSQL, and Oracle are usually used to store and manage log data. When faced with massive amounts of data, these databases use distributed databases or distributed storage methods to ensure the efficiency and scalability of the system. However, this type of database system takes up a lot of memory and CPU resources, and relies on complex libraries and software environments, which are not suitable for embedded systems with limited resources. Therefore, although these databases perform well in large and medium-sized systems, they cannot operate efficiently in embedded applications.

[0004] In embedded systems, due to limited memory and CPU resources, the lightweight database SQLite is often used. SQLite is easy to integrate, takes up few resources, and is suitable for processing small-scale log data, especially when the number of logs is less than 1 million, its storage and query efficiency is relatively high. However, as the amount of data increases (more than 1 million), the query speed of SQLite drops sharply, and may even cause the database process to crash, making it unable to cope with the storage and query requirements of massive log data. This makes SQLite have significant limitations in large-scale log management scenarios in embedded systems.

[0005] Traditional log files are stored in rows, with each row storing a log record, and the files usually exist in plain text. Although this storage method is simple, it can only be read sequentially, and the log files have no compression function, resulting in a large amount of storage space being occupied. Especially when processing a large number of logs, this row-based storage method is not only inefficient in access, but also lacks an effective retrieval mechanism, and cannot meet the needs of fast query of massive logs in embedded systems. Therefore, the traditional log file storage method is only suitable for scenarios with a small number of logs, and is extremely inefficient in large-scale log processing scenarios.

[0006] Therefore, a new method and device for storing and reading log files is needed.

[0007] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0008] In view of this, the present application provides a method and device for storing and reading log files, and provides an efficient log storage and reading mechanism suitable for embedded systems with limited resources. By optimizing hard disk writing through memory cache, it not only realizes the rapid access of massive logs, but also protects hardware devices and ensures the real-time and reliability of data.

[0009] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0010] According to one aspect of the present application, a method for storing and reading log files is proposed, the method comprising: obtaining daily operation data of the system to generate log data; combining the log data according to a preset format to generate a log file; storing the log file in a cache according to a preset directory structure; generating a log statistical information file, and saving the memory statistical information file of the day in the log statistical information file in real time in the memory; when the preset conditions are met, saving the log file in the cache to the hard disk for easy reading.

[0011] In an exemplary embodiment of the present application, it also includes: generating a file header format through a version number, a number of logs, a start time, an end time, a file sequence number, a log type, and an index offset array; generating a file body format through a log header, log content, and a log footer; and generating the preset format through the file header format and the file body format.

[0012] In an exemplary embodiment of the present application, it also includes: generating the preset directory structure according to a month, day structure and a specific naming method.

[0013] In an exemplary embodiment of the present application, obtaining system daily operation data to generate log data includes: obtaining system daily operation data; and preprocessing the daily operation data to generate the log data.

[0014] In an exemplary embodiment of the present application, the log data are combined according to a preset format to generate a log file, including: using the log data as log content; and adding a log header and a log footer to the log content.

[0015] In an exemplary embodiment of the present application, generating a log statistical information file includes: generating a daily statistical information file every day; generating a monthly statistical information file every month; and generating a memory statistical information file of the day in real time every day.

[0016] In an exemplary embodiment of the present application, it also includes: regularly updating the daily statistical information file and the monthly statistical information file through the memory statistical information file of the current day.

[0017] In an exemplary embodiment of the present application, when preset conditions are met, the log files in the cache are saved to the hard disk for easy reading, including: when the cache occupancy exceeds a threshold, the log files in the cache are saved to the hard disk; or the log files in the cache are saved to the hard disk at a scheduled time.

[0018] In an exemplary embodiment of the present application, it also includes: locating the log file to be accessed through the index offset array; and reading the log file through a forward or reverse query method.

[0019] According to one aspect of the present application, a log file storage and reading device is proposed, which includes: a data module, used to obtain system daily operation data to generate log data; a file module, used to combine the log data according to a preset format to generate a log file; a structure module, used to store the log file in a cache according to a preset directory structure; a statistics module, used to generate a log statistical information file, and save the memory statistical information file of the day in the log statistical information file in real time in the memory; a storage module, used to save the log file in the cache to the hard disk for easy reading when the preset conditions are met.

[0020] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0022] According to the log file storage and reading method and device of the present application, log data is generated by acquiring daily system operation data; the log data is combined according to a preset format to generate a log file; the log file is stored in a cache according to a preset directory structure; a log statistical information file is generated, and the memory statistical information file of the day in the log statistical information file is saved in real time in the memory; when the preset conditions are met, the log file in the cache is saved to the hard disk for easy reading, thereby providing an efficient log storage and reading mechanism suitable for embedded systems with limited resources, optimizing hard disk writing through memory cache, realizing rapid access to massive logs, protecting hardware devices, and ensuring the real-time and reliability of data.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0025] Figure 1 The present invention is a flow chart showing a method for storing and reading a log file according to an exemplary embodiment.

[0026] Figure 2-7 The diagram is a schematic diagram of a method for storing and reading a log file according to an exemplary embodiment.

[0027] Figure 8-11 The present invention is a flow chart showing a method for storing and reading a log file according to another exemplary embodiment.

[0028] Fig.12 The present invention is a block diagram of a log file storage and reading device according to an exemplary embodiment.

[0029] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment.

[0030] Fig.14 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0032] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0035] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0036] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0037] The technical abbreviations involved in this application are explained as follows:

[0038] MySQL: It is a relational database management system developed by MySQL AB of Sweden and is a product of Oracle. MySQL is one of the most popular relational database management systems. In terms of WEB applications, MySQL is one of the best RDBMS (Relational Database Management System) application software.

[0039] Oracle: also known as OracleDatabase, also known as Oracle RDBMS, or simply Oracle. It is a relational database management system of Oracle Corporation. It is a product that has always been in a leading position in the database field. It can be said that the Oracle database system is a popular relational database management system in the world. The system has good portability, is easy to use, and has powerful functions. It is suitable for all kinds of large, medium and small microcomputer environments. It is a highly efficient, reliable, and high-throughput database solution.

[0040] PostgreSQL: is a free software object-relational database management system (ORDBMS) with a very complete set of features. It is an object-relational database management system based on POSTGRES version 4.2 developed by the Department of Computer Science at the University of California. Many of POSTGRES's leading concepts only appeared in commercial website databases at a relatively late time. PostgreSQL supports most of the SQL standard and provides many other modern features, such as complex queries, foreign keys, triggers, views, transaction integrity, multi-version concurrency control, etc.

[0041] SQLite: is a lightweight database and a relational database management system that complies with ACID. It is contained in a relatively small C library. It is designed for embedded use and has been used in many embedded products. It takes up very little resources. In embedded devices, only a few hundred KB of memory may be enough. It can support mainstream operating systems such as Windows / Linux / Unix, and can be combined with many programming languages, such as Tcl, C#, PHP, Java, etc., and has an ODBC interface. It is also faster than Mysql and PostgreSQL, two world-renowned open source database management systems.

[0042] Figure 1 1 is a flow chart of a method for storing and reading a log file according to an exemplary embodiment. The method 10 for storing and reading a log file at least includes steps S102 to S110.

[0043] like Figure 1As shown, in S102, daily operation data of the system is obtained to generate log data. For example, daily operation data of the system may be obtained; and the daily operation data may be preprocessed to generate the log data.

[0044] More specifically, daily operation data can be continuously monitored and collected to generate corresponding log data. These operation data can include various events generated within the system, operation commands, user interaction information, system resource status changes, etc. For example, the server may record operation information such as process startup, user login, file access, network request, etc. After being collected, these raw data will be pre-processed, including formatting, denoising, deduplication and other operations, to generate structured log data. Log data can contain information such as timestamp, event type, operation description, etc., to provide basic data for subsequent storage and query.

[0045] In a specific embodiment, for a single log or record (generally an information string containing a timestamp and a string description, etc.), in order to reduce the space occupied by the log or to hide the plain text information, the log may be compressed, encrypted, encoded, etc. during preprocessing. For example, it is recommended to perform protobuf encoding and serialization processing on the log. Of course, it is also possible to save the original plain text information directly without any processing.

[0046] In S104, the log data is combined according to a preset format to generate a log file. For example, the log data can be used as log content; and a log header and a log footer are added to the log content.

[0047] After the log data is generated, the system will combine it in a predetermined format to form a standardized log file. In this process, the log data is used as the content of the log file, and each log record is attached with a log header and a log footer. The log header usually contains the basic metadata of the log, such as the log number, generation time, log source, etc.; the log footer is used to encapsulate the end mark information of the log for subsequent forward or reverse queries.

[0048] In S106, the log files are stored in the cache according to the preset directory structure. The generated log files will first be stored in the system's cache. The files in the cache are organized and managed according to the pre-designed directory structure, usually with year, month and day as the directory hierarchy. For example, the log files generated every day will be saved in the directory corresponding to that day to avoid storing too many files in a single directory, which will lead to a decrease in access efficiency. The use of cache can effectively improve the immediacy of data storage, reduce frequent read and write operations on the hard disk, and extend the life of the hard disk. At the same time, this mechanism can be flexibly adjusted according to conditions such as file size and number of logs to ensure that the files in the cache can be efficiently managed and retrieved.

[0049] In S108, a log statistical information file is generated, and the memory statistical information file of the day in the log statistical information file is saved in real time in the memory. For example, a daily statistical information file may be generated every day; a monthly statistical information file may be generated every month; and a memory statistical information file of the day may be generated in real time every day. For example, the daily statistical information file and the monthly statistical information file may be updated regularly through the memory statistical information file of the day every day.

[0050] While storing logs, log statistics files can also be generated. These files are used to record the overall information of log data for quick query and analysis. For example, the daily statistics file will record the number of log files generated on that day, the total amount of logs, the type of logs, and other information, while the monthly statistics file will record the monthly log summary. In order to reduce the frequent writing to the hard disk, the system will save the real-time data of these statistics in the memory and achieve fast access to the data through memory caching. The daily log statistics will be synchronously updated to the daily statistics file and monthly statistics file in the hard disk at a specific time period (such as early morning every day) to ensure the accuracy and integrity of the data.

[0051] In S110, when a preset condition is met, the log file in the cache is saved to the hard disk for easy reading. For example, when the cache occupancy exceeds a threshold, the log file in the cache is saved to the hard disk; for example, the log file in the cache is saved to the hard disk at a fixed time.

[0052] In order to balance system performance and data persistence, the system sets trigger conditions for writing cache to the hard disk. Log files in the cache can be saved to the hard disk when one of the following two conditions is met:

[0053] When the amount of log data in the cache exceeds the preset maximum value, the system will automatically write the log files in the cache to the hard disk in batches to free up cache space and prevent cache overflow.

[0054] You can also use a timing strategy to regularly save the log files in the cache to the hard disk, for example, write them once every certain period of time (such as 3 seconds or 5 minutes). This can ensure the persistent storage of data and avoid excessive system pressure caused by excessive cache usage.

[0055] Through this batch and scheduled writing mechanism, the system can not only ensure the reliable storage of log data, but also reduce frequent hard disk write operations and improve overall performance.

[0056] In one embodiment, the log file to be accessed can be located through the index offset array; the log file can also be read through forward or reverse query. The index offset array records the offset position of the key log in the file, and can quickly locate the target log according to the log sequence number or query condition without traversing the entire file. In addition, both forward and reverse query methods are supported. The forward query retrieves the log in chronological order, and the reverse query can quickly locate the most recent log record. This design greatly improves the retrieval efficiency of log files, especially in the scenario of massive log data, and can quickly respond to query requests to meet the needs of different business scenarios.

[0057] According to the log file storage and reading method of the present application, log data is generated by acquiring daily system operation data; the log data is combined according to a preset format to generate a log file; the log file is stored in a cache according to a preset directory structure; a log statistical information file is generated, and the memory statistical information file of the day in the log statistical information file is saved in real time in the memory; when the preset conditions are met, the log file in the cache is saved to the hard disk for easy reading, thereby providing an efficient log storage and reading mechanism suitable for embedded systems with limited resources, optimizing hard disk writing through memory cache, realizing rapid access to massive logs, protecting hardware devices, and ensuring the real-time and reliability of data.

[0058] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0059] In some embedded systems, although system resources (such as memory and CPU) are limited, it is still necessary to store a large number of logs or record data at the terabyte level. Since existing database systems are difficult to meet this demand in resource-constrained environments, the present invention proposes a specially designed efficient file storage and fast reading mechanism. This mechanism can achieve efficient storage and fast access of large-scale logs while occupying very few system resources.

[0060] The directory structure of the log file is designed as follows:

[0061] 202310 (monthly folder) ├──202310 (log statistics file for the current month)

[0062] ├──20231011 (statistical information file for that day)

[0063] ├──2023101100001 (the first log file of the day)

[0064] ├──2023101100002 (the second log file of the day)

[0065] ├──2023101100003 (the third log file of the day)

[0066] ├──20231012 (statistical information file of the day)

[0067] ├──2023101200001 (the first log file of the day)

[0068] ├──2023101200002 (the second log file of the day)

[0069] ├──2023101200003 (the third log file of the day)

[0070] Log files are saved monthly with the year and month (such as 202310) as the folder name, and a new folder is created every month to prevent too many files in the same folder from reducing the access rate. The log file name starts with the date, and the file number is increased to prevent too many files on the same day. For example, the first log file saved on the day is named 2023101100001, which means the first log file on October 11, 2023, and 2023101100002 means the second log file on October 11, 2023. This allows you to quickly find the nth file on any day.

[0071] Each log file consists of two parts: the file header and the file body. The file header format is as follows Figure 2 As shown, it contains fields such as version number, log quantity, log start time, end time, file sequence number, log type, index offset array, etc.

[0072] like Figure 3 As shown, the version number is used to distinguish when the format has changed and is also used to verify the validity of the file.

[0073] The number of logs records the number of logs in the log file.

[0074] Start time, the time when the first log of the file is generated. It can be used to quickly determine when querying by time period without traversing the entire file. It can also be used to verify the correctness of the date in the file name, whether the file has been accidentally modified, etc.

[0075] End time, the time when the last log of the file was generated. When querying by time, you can quickly determine the result without traversing the entire file.

[0076] The file sequence number is used to identify the log file of the day, starting from 1. It can be used to verify the sequence number in the file name.

[0077] The log type is used to identify what type of log the file stores when necessary, such as security log, traffic log, operation log, etc. Generally, each log file only stores one type of log.

[0078] The index offset array is the offset of the key position log, which is used to quickly locate and read the log content. For example, if the file stores 1 million logs, the array member [0] stores the offset of the 10,000th log, the array member [1] stores the offset of the 20,000th log, and the array member [2] stores the offset of the 30,000th log. The size of the array can be determined according to actual needs. In this way, the corresponding log can be quickly located according to the offset, without having to parse and traverse the full text of each log.

[0079] The file body consists of multiple logs. Each log contains three parts: log header, log content, and log tail. The format of log header and log tail is the same. Figure 4 As shown:

[0080] Datetime is a time. You can quickly search and skip logs in the corresponding time period by the log head and tail time without parsing / decoding the log content.

[0081] The log content length indicates the actual buffer length occupied by the log content, which is used to extract and parse this log or quickly skip this log. The log content can be original text or binary data after encoding, compression, encryption, serialization, etc.

[0082] Key index fields are fields that need to be frequently queried, such as ID, IP, etc., to facilitate quick retrieval without parsing logs. Generally, integer fields are recommended.

[0083] The log type is used to indicate what type of log is stored in the log, such as security log, traffic log, operation log, etc. When you need to save multiple types of logs in one log file, you can add this field for quick retrieval or skipping.

[0084] The fields in the log header and log footer are generally defined as integer fields with fixed length, occupying less memory and facilitating fast matching.

[0085] The same format of the log header and tail makes it easy to read the log from the beginning of the file to the end of the file, or from the end of the file to the beginning of the file as needed. This is more flexible and speeds up the file reading speed under various forward and reverse order conditions.

[0086] The number of logs saved in each file is the same, set to n. For example, the following assumes that each file saves 1 million logs. After reaching 1 million logs, another file is created to save the logs.

[0087] More specifically, Figure 5 As shown in the figure, the daily statistics file is used to save the total number of logs per day and other statistics. For example, the 20231011 file saves the log statistics for that day. To read the statistics for any day, you can directly read the file using the date file name.

[0088] Date, such as 20231011, is consistent with the name of the log statistics file, indicating that the statistics for that day are stored.

[0089] The number of log files on the day indicates how many log files there are on that day. You can quickly determine the name of any log file on that day. Combining the date and sequence number, you can quickly locate any log file.

[0090] The total number of logs for the day indicates the total number of logs for the day, which is convenient for quick statistics, retrieval or skipping of the day's logs. To avoid damaging the disk, the total number of logs for the day (the latest day) is saved in memory and will not be written into the daily statistics file until midnight.

[0091] More specifically, Figure 6 As shown in the figure, the monthly statistics file is used to save the total number of logs per month and other statistical information. For example, the 202310 file saves the log statistics for October 2023. To read the statistics of any month, you can directly read the file using the year and month file names.

[0092] Date, such as 202310, is consistent with the name of the log statistics file for the current month, indicating that the statistics for that month are stored.

[0093] The total number of monthly logs indicates the total number of logs for that month, which is convenient for quick statistics, retrieval or skipping of logs for that month. To avoid damaging the disk, the total number of historical months is the actual total number of that month. The total number of the current month does not include the latest number of the day. The total number of the day will be updated from the memory into the monthly statistics file in the early morning.

[0094] More specifically, Figure 7 As shown in the figure, the memory statistics file of the current day is used to save the log statistics of the current day in the memory in real time (most of the fields are the same as those of the daily statistics file).

[0095] The data in the memory is updated in real time, and the information is updated to the daily statistics file and monthly statistics file every morning. Theoretically, the total number of logs on the day = the total number of logs in the latest log file + (the number of log files on the day - 1) * 1 million.

[0096] Figure 8 The present invention is a flow chart showing a method for storing and reading a log file according to another exemplary embodiment. Figure 8The process 80 shown is a detailed description of writing logs to the hard disk.

[0097] like Figure 8 As shown, in S802, the log starts to be written to the hard disk.

[0098] In S804, it is determined whether the current time and the latest log file are on the same day.

[0099] In S806, the memory statistical information is updated to the monthly statistical information file.

[0100] In S808, it is determined whether the number of file logs reaches a threshold, where the threshold may be 1 million or other set values.

[0101] In S810, the file header information is updated and the file is closed, and a new log file is created.

[0102] In S812, the log is written into the file, and the log statistics information in the memory is updated.

[0103] In order to prevent the hard disk from being damaged due to frequent writing to the same hard disk location, optimization measures can be taken, for example, when updating the number of logs. Since frequent updates of the number of logs in the file header require frequent hard disk write operations, this operation will accelerate the wear of the hard disk. To solve this problem, the log statistics of the day can be saved in the memory in real time. When the log file reaches 1 million records and a new file is ready, the file header of the current log file is updated for the last time. More specifically, at 0:00 a.m. every day, the log statistics in the memory can be updated to the daily statistics file of the day and the monthly statistics file of the month.

[0104] In addition, the log data will be stored in the memory cache first. When the cache is about to be full or reaches a certain time period (for example, every 3 seconds), the log data in the cache can be saved in batches to the hard disk. This batch storage method reduces the number of hard disk writes, improves storage efficiency, and extends the service life of the hard disk.

[0105] Fig. 9 The present invention is a flow chart showing a method for storing and reading a log file according to another exemplary embodiment. Fig. 9 The process 90 shown is a detailed description of restoring log statistics after restart.

[0106] like Fig. 9 As shown, in S902, configuration recovery starts.

[0107] In S904, the latest log file information of the hard disk is read.

[0108] In S906 , it is determined whether the latest log file is a valid log file.

[0109] In S908, the invalid log file is deleted.

[0110] In S910, the number of files is read and updated into the memory log statistical information.

[0111] In S912, it is determined whether the latest log file is the log file of the day.

[0112] In S914, the number of file logs and the number of files are read and updated into the memory log statistics information.

[0113] In S916, the file log quantity is read, its log header information is updated, the memory information date is set, and the quantity is cleared.

[0114] In S918, end.

[0115] When the device restarts unexpectedly, the number of logs saved in the memory for the day will be lost, so it is necessary to restore the log statistics. When the system restarts, it first checks the file header, log header, log footer, and cache length to determine whether the log file is valid. If the log file is invalid, the system will delete the file and update the statistics in the memory; if the file is valid, the system will read the number of logs saved to the hard disk for recovery. Each log file can store up to 1 million records. The latest log file determines the actual number of saved logs by traversing its content, and then updates the number to the memory and the log file header information.

[0116] Fig.10 The present invention is a flow chart showing a method for storing and reading a log file according to another exemplary embodiment. Fig.10 The process 100 shown is a detailed description of reading log data.

[0117] like Fig.10 As shown, in S1002, the monthly statistical information file is traversed to read the monthly total.

[0118] In S1004, the total number of logs per month is summed.

[0119] In S1006, the sum of the total number of monthly log statistics is added to the total number of logs on the day.

[0120] In S1008, the total number of logs is obtained.

[0121] When querying logs, you usually need to quickly obtain the total number of logs for paging display. The total number of logs in the monthly statistics file is read and summed, and then added to the total number of logs in the memory for the day (to protect the hard disk device, the latest total number of logs for the day is stored in the memory), which is the total number of historical logs.

[0122] Fig.11The present invention is a flow chart showing a method for storing and reading a log file according to another exemplary embodiment. Fig.11 The process 110 shown is a detailed description of reading log data.

[0123] like Fig.11 As shown, in S1102, the number of months is read month by month, and the specific month is determined according to the offset.

[0124] In S1104, the number of daily logs in the month is read and the specific day is determined according to the offset.

[0125] In S1106, the total number of log files and the number of the last file log are read, and the specific log file is determined according to the offset.

[0126] In S1108, the log file and offset are read, and the log is read.

[0127] Page turning query is the most commonly used operation. For example, if there are 10 million logs in total, and each page displays 100 lines of log records, there are 100,000 pages in total. To view page 1000, you need to offset 1000×100 (i.e. 100,000) logs, and then read 100 logs in succession. The following is a flow on how to quickly turn to any page to query logs. Since each file is limited to saving 1 million logs, you can use the formula to quickly locate a specific log file for reading.

[0128] This application designs an efficient log file storage and query mechanism with the following features:

[0129] Efficient query: fast forward and reverse query of log records is realized, and each file saves a fixed number of logs, and the specific file is quickly located through offset calculation.

[0130] Memory statistics: The number of logs and statistical information for the day are saved in real time in the memory to avoid frequent writing to the hard disk, protect the hard disk and ensure the real-time accuracy of the data.

[0131] Recovery mechanism: When a process exits unexpectedly, the memory statistics for that day can be updated and restored based on the saved log files.

[0132] Index optimization: The log file header stores the offsets of different index intervals, so that specific intervals can be quickly located during query without traversing the entire file.

[0133] Quantitative storage: Each log file stores the same number of records, which is convenient for total log statistics and allows quantitative deletion based on date or file.

[0134] Time verification: The file header stores the time of the first and last logs, which is convenient for verification with the file name, and quickly determines whether the file is within the query time range, reducing the need for traversal.

[0135] Bidirectional reading: Each log has a unified format, which is convenient for flexible reading from front to back or from back to front, improving reading efficiency.

[0136] Condition matching: The log header and footer contain time and key index fields, supporting fast condition matching without parsing the entire log.

[0137] These designs enable efficient management and query of massive log data while ensuring data security and integrity.

[0138] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are executed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0139] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0140] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0141] Fig.12 FIG. 1 is a block diagram of a log file storage and reading device according to an exemplary embodiment. Fig.12 As shown, the log file storage and reading device 120 includes: a data module 1202 , a file module 1204 , a structure module 1206 , a statistics module 1208 , and a storage module 1210 .

[0142] The data module 1202 is used to obtain system daily operation data to generate log data;

[0143] The file module 1204 is used to combine the log data according to a preset format to generate a log file;

[0144] The structure module 1206 is used to store the log files in the cache according to the preset directory structure;

[0145] The statistics module 1208 is used to generate a log statistics information file, and save the memory statistics information file of the day in the log statistics information file in real time in the memory;

[0146] The storage module 1210 is used to save the log files in the cache to the hard disk for easy reading when the preset conditions are met.

[0147] According to the log file storage and reading device of the present application, log data is generated by acquiring daily system operation data; the log data is combined according to a preset format to generate a log file; the log file is stored in a cache according to a preset directory structure; a log statistical information file is generated, and the memory statistical information file of the day in the log statistical information file is saved in real time in the memory; when the preset conditions are met, the log file in the cache is saved to the hard disk for easy reading, thereby providing an efficient log storage and reading mechanism suitable for embedded systems with limited resources, optimizing hard disk writing through memory cache, realizing fast access to massive logs, protecting hardware devices, and ensuring the real-time and reliability of data.

[0148] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment.

[0149] Refer to the following Fig.13 1300 according to this embodiment of the present application is described. Fig.13 The electronic device 1300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Fig.13 As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), a display unit 1340, etc.

[0151] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 1310 can perform the steps shown above.

[0152] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 13201 and / or a cache memory unit 13202 , and may further include a read-only memory unit (ROM) 13203 .

[0153] The storage unit 1320 may also include a program / utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0154] Bus 1330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0155] The electronic device 1300 may also communicate with one or more external devices 1300' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) to enable a user to interact with the electronic device 1300, and / or any device (e.g., routers, modems, etc.) that the electronic device 1300 can communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface 1350. In addition, the electronic device 1300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1360. The network adapter 1360 may communicate with other modules of the electronic device 1300 via the bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Fig.14 As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0157] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0158] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0159] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0160] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium realizes the following functions: obtaining system daily operation data to generate log data; combining the log data according to a preset format to generate a log file; storing the log file in a cache according to a preset directory structure; generating a log statistical information file, and saving the memory statistical information file of the day in the log statistical information file in real time in the memory; when the preset conditions are met, saving the log file in the cache to the hard disk for easy reading.

[0161] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0162] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0163] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A method for storing and reading a log file, characterized in that: include: Obtain system daily operation data to generate log data; Combining the log data according to a preset format to generate a log file; Store log files in the cache according to the preset directory structure; Generate a log statistics file, and save the memory statistics file of the day in the log statistics file in real time in the memory; When the preset conditions are met, the log files in the cache are saved to the hard disk for easy reading.

2. The method according to claim 1, characterized in that Also includes: Generate the file header format through version number, log quantity, start time, end time, file sequence number, log type, and index offset array; Generate the file body format through the log header, log content, and log footer; The preset format is generated by using the file header format and the file body format.

3. The method according to claim 1, characterized in that Also includes: The preset directory structure is generated according to the month, day structure and specific naming method.

4. The method according to claim 1, characterized in that Obtain system daily operation data to generate log data, including: Obtain system daily operation data; The daily operation data is preprocessed to generate the log data.

5. The method according to claim 1, characterized in that Combining the log data according to a preset format to generate a log file includes: Using the log data as log content; Add a log header and a log footer to the log content.

6. The method according to claim 1, characterized in that Generates log statistics files, including: Generate daily statistics files every day; Generate monthly statistics files every month; Generates the memory statistics file of the day in real time every day.

7. The method according to claim 6, characterized in that Also includes: The daily statistical information file and the monthly statistical information file are updated regularly every day through the memory statistical information file of the day.

8. The method according to claim 1, characterized in that When the preset conditions are met, the log files in the cache are saved to the hard disk for easy reading, including: When the cache occupancy exceeds the threshold, the log files in the cache are saved to the hard disk; or Save the log files in the cache to the hard disk at regular intervals.

9. The method according to claim 1, characterized in that Also includes: Locate the log file to be accessed through the index offset array; The log file is read by forward or reverse query.

10. A log file storage and reading device, characterized in that: include: Data module, used to obtain system daily operation data to generate log data; A file module, used for combining the log data according to a preset format to generate a log file; A structure module is used to store log files in a cache according to a preset directory structure; The statistics module is used to generate log statistics information files and save the memory statistics information files of the day in the log statistics information files in real time in the memory; The storage module is used to save the log files in the cache to the hard disk for easy reading when the preset conditions are met.

Images