Method, device and equipment for converting log data into storage

Abstract

The invention discloses a method, a device and equipment for converting log data into storage, and the method comprises the following steps: generating a dynamic log file in a memory file system at regular time by utilizing a system log service tool of a Linux operating system; using the system log service tool to read log data to a latest dynamic log file; calling a log merging script of a user mode of the Linux operating system, when it is determined that log file to storage conditions are met by executing the log merging script, reserving at least one dynamic log file according to the sequence of generation time from back to front, and merging the rest dynamic log files according to the sequence of time from front to back; and transferring and storing the log file obtained after merging into a disk file system. According to the method, the problem that log data may be lost due to the fact that log is converted into storage through logate and Rsyslod is used for collecting files into new log files in an asynchronous mode in the prior art is solved.

Description

Background Art

[0002] In the Linux operating system, the system log service tool Rsyslogd is used for log collection and log filtering. The input source is the kernel log ring buffer, and the output destination is the log file, i.e., the messages file. The default configuration file of Rsyslogd is configured in the specified directory. The file name and path of the messages file can be specified or modified.

[0003] Log cutting means that when a log file reaches the set trigger condition, it is cut / split, similar to truncation, and the original log file with a larger capacity is "robbed" and transferred to another log file for archiving. The logs generated after this moment continue to be output to the log file whose file header is reset to 0. This process is also commonly known as log rotation.

[0004] Currently, the Linux operating system uses the encapsulated system framework component logrotate to transfer files when the monitoring reaches the trigger condition. However, since the system framework component logrotate provides users with only configuration file modification and shell script callback interfaces, it cannot find a suitable time to do log dumping. It is an asynchronous operation with the system log service tool Rsyslogd to collect files into new log files. Therefore, log data may be lost, and kernel log data loss will seriously affect the normal operation of the Linux operating system. Summary of the invention

[0005] The purpose of this application is to provide a log data transfer storage method, device and equipment to solve the problem that the existing log transfer storage using logrotate and Rsyslogd collecting files into new log files are asynchronous operations, and there may be a situation where log data is lost.

[0006] In a first aspect, an embodiment of the present application provides a log data transfer storage method, the method comprising:

[0007] Use the system log service tool of the Linux operating system to generate dynamic log files in the memory file system at regular intervals;

[0008] Using the system log service tool, read the log data into the latest dynamic log file;

[0009] Calling a log merging script in the user state of the Linux operating system, and when it is determined by executing the log merging script that the log file transfer storage condition is met, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back;

[0010] The merged log files are transferred to the disk file system.

[0011] The embodiment of the present application changes the way the system log service tool rsyslog generates log files, from writing log data only to a single large log file, to continuously and dynamically generating small dynamic log files, each of which is used to store log data for a period of time. By adding a merge script file in the user state, when it is determined that the log transfer storage conditions are met, newer dynamic log files are reserved, and the remaining dynamic logs are merged and transferred to the disk file system. Therefore, the problem of rsyslogd and logrotate cooperation in the related technology does not exist, ensuring that log data will not be lost, thereby ensuring the normal operation of the Linux operating system.

[0012] In some possible embodiments, at least one dynamic log file is reserved in the order of generation time from the last to the first, and the remaining dynamic log files are merged in the order of time from the first to the last, including:

[0013] The latest dynamic log file is reserved in the order of generation time from the latest to the earliest, and all other dynamic log files are merged in the order of generation time from the earliest to the latest.

[0014] In the embodiment of the present application, dynamic log files are merged according to the order in which they are generated, thereby retaining the original order in which the log data is arranged.

[0015] In some possible embodiments, determining whether the log file transfer storage condition is satisfied by executing the log merging script includes:

[0016] The monitoring time from the last merging of dynamic log files reaches the set time, and the log file transfer storage conditions are determined to be met; or

[0017] According to the comparison result of the size of the currently generated dynamic log file and the set file threshold, it is determined that the log file transfer storage condition is met.

[0018] In the embodiment of the present application, two methods can be used to determine whether the log transfer storage conditions are met. One is to set a corresponding time length, and perform dynamic log file merging and transfer storage when a certain time length is reached. The setting of the time length ensures that a sufficient amount of log data can be accumulated. The other method is to determine whether the continuously generated dynamic log files form a file of sufficient size, thereby avoiding frequent log transfer storage.

[0019] In some possible embodiments, a system log service tool of a Linux operating system is used to periodically generate a dynamic log file in a memory file system, including:

[0020] Using the system log service tool of the Linux operating system, determine that when the interval from the last dynamic log file generation reaches a set interval, the latest dynamic log file is used as the cut log file;

[0021] Generate a new dynamic log file to store the log data read subsequently.

[0022] In the embodiment of the present application, a system log service tool is used to continuously generate new dynamic log files by cutting files, and the functions originally performed by the system framework component are transferred to the system log service tool for implementation, thereby avoiding the problem of the system framework component cutting and transferring log files and the system log service tool writing log data into the log file out of sync.

[0023] In some possible embodiments, when it is determined by executing the log merging script that the log file transfer storage condition is met, at least one dynamic log file is reserved in order of generation time from back to front, and the remaining dynamic log files are merged in order of time from front to back, including:

[0024] By executing the log merging script, each time a cut log file is generated, the cut log file is spliced ​​with the previous cut log file;

[0025] When it is determined that the size of the spliced ​​cutting log file exceeds the set file threshold, it is determined that the log file transfer storage condition is met.

[0026] In the process of generating dynamic log files, the embodiment of the present application continuously splices the cut dynamic log files, and determines that the log transfer storage condition is met when the spliced ​​log files exceed a threshold, so that when merging dynamic log files, the spliced ​​dynamic log files are directly merged.

[0027] In some possible embodiments, the initially generated dynamic log file is named as the first file name, and the subsequently generated dynamic log file name includes the first file name plus an extension, wherein the extension carries timing information;

[0028] The merged log file uses the second file name, and the name of the latest dynamic log file reserved after the merger is updated to the first file name.

[0029] When naming a dynamic log file, the embodiment of the present application adds an extension to the log file and the extension carries the timing information of the dynamic log file generation. Therefore, the timing of the dynamic log file generation can be determined according to the extension, and the dynamic log file is renamed according to the same mechanism as the log file is transferred and stored.

[0030] In some possible embodiments, the latest dynamic log file is reserved in the order of generation time from the latest to the latest, and all other dynamic log files are merged in the order of time from the earliest to the latest, including:

[0031] In order of generation time from the latest to the earliest, the latest dynamic log file is reserved, and the generation sequence of the remaining dynamic log files is determined according to their extensions.

[0032] The log files are merged in a time-to-time order according to the generation sequence, the merged log files are given the second file name, and the name of the reserved latest dynamic log file is updated to the first file name.

[0033] The embodiment of the present application can determine the generation sequence according to the extension of the dynamic log file, so as to merge the files according to the generation sequence, and after the files are merged, the reserved dynamic log file name is named according to the initial dynamic log file naming method.

[0034] In a second aspect, an embodiment of the present application provides a log data storage device, the device comprising:

[0035] A dynamic log file generation module is used to utilize the system log service tool of the Linux operating system to periodically generate dynamic log files in the memory file system;

[0036] A log data reading module, used to read log data into the latest dynamic log file using the system log service tool;

[0037] The log file merging module is used to call the log merging script in the user state of the Linux operating system. When it is determined that the log file transfer storage condition is met by executing the log merging script, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back;

[0038] The log transfer module is used to transfer the merged log files to the disk file system.

[0039] In the third aspect, another embodiment of the present application also provides a log data transfer and storage device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any log data transfer and storage method provided in the embodiment of the present application.

[0040] In a fourth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any log data conversion and storage method provided in the embodiments of the present application.

[0041] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 This is a schematic diagram of the process of using rsyslogd to read log data and store it in the messages file in the related technology;

[0044] Figure 2 A schematic diagram of the log cutting process for the related technology;

[0045] Figure 3 A schematic diagram of a log cutting process of a disk file system in the related art;

[0046] Figure 4 A schematic diagram of a log cutting process of a memory file system in a related art;

[0047] Figure 5 A flow chart of a log data transfer and storage method provided according to an embodiment of the present application;

[0048] Figure 6 A detailed flow chart of a log data transfer and storage method according to an embodiment of the present application;

[0049] Figure 7 This is a structural diagram of a log data transfer storage device according to an embodiment of the present application;

[0050] Figure 8 The present invention is a structural diagram of a log data transfer storage device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0052] In the Linux operating system, there are two input sources for log data generation. One input source is the kernel log data generated by the kernel prink function, and the other input source is the user-mode log data generated by executing the user-mode logger command through the device interface. The device interface can be dynamically generated during process management, file management, and device management in the Linux operating system. The rsyslogd of the Linux operating system is a user-mode process responsible for continuously retrieving the log data from the cache and storing it in the messages file. Figure 1 As shown, the input part includes the above-mentioned input source. The log data generated by the above-mentioned input source is queued after being pre-processed by the system log service tool rsyslogd. The corresponding log write request is parsed by calling the HTTP message parser and responded to. Specifically, the queued log data is filtered by the filter engine and then added to the corresponding action queue action aueue. The log data is output to the messages file through the action queue action aueue through the action processor Action Processor.

[0053] Log cutting means that when the message files reach the set trigger conditions, such as according to a certain time period (daily), or according to the file size (100MB), they are cut / split, similar to truncation, and the original large log file is "robbed" and transferred to another log file for archiving. The logs generated after this moment continue to be output to the new log file whose file header is reset to 0. This process is also commonly known as log rotation. The encapsulated system framework component logrotate currently provided by the Linux operating system can automatically rotate, compress, delete and mail log files according to the configuration file to reduce the size of the log file and maintain the normal operation of the system. Figure 2The example log rolling process is shown. Specifically, after the Linux operating system starts the application, the log file is 0MB at initialization. After one day of running, the log file reaches 500MB, meeting the threshold for cutting the log file. The log file is cut, the existing 500MB log file is transferred and a new 0MB log file is created. The file name of the new log file is the same as the file name of the cut log file.

[0054] After Rsyslogd collects kernel log data and user-mode log data, the storage destination, i.e., the log file, may be in the disk file system or in the memory file system. If rsyslogd reads the generated log data into the log file directly in the disk file system, when the log file splitting condition is met, it only needs to rename the transferred log file and create a new log file with the same file name as the original one, and there is no need for log dumping. That is, logrotate creates a new file, and before rsyslogd receives the notification, it still writes the log to the previous log file, regardless of whether the file name is renamed. Figure 3 The following is a schematic diagram of the log rolling process of the output directory of log data in the disk file system scenario, which mainly includes:

[0055] Step 1: The Linux operating system continuously generates log data while running the APP. rsyslogd stores the generated log data in the disk file system. When Logrotate determines that the conditions for file cutting are met, it starts cutting the log file (mv old log file). Specifically, Logrotate renames the old log file.

[0056] Step 2: Logrotate creates a new empty log file with the same name as the old log file.

[0057] Step 3, Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification;

[0058] Step 4: rsyslogd changes the handle of the log file from pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data is written to the new log file.

[0059] If the log file is in the memory file system, rsyslogd will read the generated log data into the memory log file. Therefore, when the log file cutting conditions are met, the memory log file needs to be transferred to the disk file system and a new log file with the same file name as the original one needs to be created. That is, in the scenario where the log output directory is in the memory file system, you need to add a "log dump". That is, move the renamed log file from the memory file system to the disk file system. Executing the mv command across file systems essentially copies the file first and then deletes the original file. Rsyslogd and logrotate rely on sending a SIGHUP signal to switch the file's logrotate to create a new file. Before rsyslogd receives the notification, it still writes the log to the previous log file, regardless of whether the file name has been transferred. For example Figure 4 The following is a schematic diagram of the log rolling process of the output directory of log data in the memory file system scenario, which mainly includes:

[0060] Step 1: The Linux operating system continuously generates log data while running the APP. rsyslogd stores the generated log data in the memory file system. When Logrotate determines that the conditions for file cutting are met, it starts to cut the log file (mv the old log file). Specifically, Logrotate renames the old log file.

[0061] Step 2: Logrotate creates a new empty log file with the same name as the old log file.

[0062] Step 3, Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification;

[0063] In step 4.1, rsyslogd changes the handle of the log file from pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data will be written to the new log file.

[0064] In step 4.2, rsyslogd transfers the renamed log files to the disk file system.

[0065] For log files, the transfer of log files is performed by Logrotate, which is equivalent to the log file reading end, and the output of newly generated log data to log files is completed by rsyslogd, which is equivalent to the log file writing end. However, the writing end and the reading end are not synchronized at present. At present, when Logrotate detects that the log cutting conditions are met, it only sends a suspended SIGHUP signal to rsyslogd, and there is no handshake between the two. Therefore, the above steps 4.1 and 4.2 are performed concurrently, and there is no guarantee that step 4.2 will be executed after step 4.1. In this way, it is possible that before rsyslogd has successfully switched to writing log data to the new log file, Logrotate has already executed the cutting and transfer of log files, so that the log data cannot be correctly written to the log file, resulting in the loss of log data.

[0066] Logrotate is a system framework component encapsulated in the Linux operating system. It can automatically rotate, compress, delete and mail log files according to the configuration file to reduce the size of the log file and maintain the normal operation of the system. The interface provided to the user is only the configuration file modification and shell script callback interface. Through the interface provided to the user, it is impossible to find the time point for switching between the old and new logs, so it is impossible to find a suitable time to do log dump. It is impossible to complete the synchronization of step 4.1 and step 4.2 through the interface provided to the user by Logrotate, that is, the process of Logrotate cutting and transferring log files cannot be realized, and it is strongly coupled with the process of rsyslogd writing log data to the log file.

[0067] In view of the fact that log data transfer and storage using logrotate in the related art and Rsyslogd collecting files into new log files are asynchronous operations, log data may be lost. In this embodiment of the present application, a log data transfer and storage method is provided, which is applied to the Linux operating system, such as Figure 5 As shown, the method includes:

[0068] Step 501, using the system log service tool of the Linux operating system to regularly generate a dynamic log file in the memory file system;

[0069] In the embodiment of the present application, the system log service tool can be, but is not limited to, the system log service tool rsyslog provided by the current Linux operating system. Rsyslog is a powerful system log service tool. The main functions are log collection and log filtering. The specific process of rsyslog writing log data to the log file is: the log data generated by the above-mentioned input source is queued after pre-processing by the system log service tool rsyslogd, and the corresponding log write request is parsed and responded by calling the HTTP message parser parser, specifically filtering the queued log data through the filter engine and adding them to the corresponding action queue action aueue respectively, and outputting the log data to the messages file through the action processor Action Processor through the action queue action aueue.

[0070] There are two input sources for the generation of log data. One input source is the kernel log data generated by the kernel prink function. rsyslog writes the kernel log data to the log file in the memory, i.e., the messages file. The other input source is the user-mode log data generated by executing the user-mode logger command through the device interface. rsyslog collects the user-mode log data and the kernel log data and writes them to the messages file of the disk file system or the messages file of the memory file system. The transfer and storage of log data refer to the scheme of the related technology, that is, when Logrotate determines that the conditions for file cutting are met, it starts to cut the log file (mv the old log file), specifically, Logrotate renames the old log file; Logrotate creates a new empty log file with the same name as the original old log file name; Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification; rsyslogd changes the handle of the log file from originally pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data is written to the new log file.

[0071] Another input source for generating log data is the kernel function. The embodiment of the present application is mainly used to solve the problem of asynchronous cutting, storage and writing of log files corresponding to log data. The system log service tool of the Linux operating system is used to read log data into log files, including the system log service tool of the Linux operating system reading log data into the log file in the memory, namely, the messages file. The messages file storing the log data is configured under the kernel directory, and the specific location can be flexibly configured as needed.

[0072] The embodiment of the present application utilizes the system log service tool of the Linux operating system to periodically generate dynamic log files in the memory file system. Timing specifically refers to generating a dynamic log file at a certain interval. The method provided by the embodiment of the present application is different from the rsyslogd in the related art that generates a large single log file and cuts it down when the conditions are met. The embodiment of the present application periodically and continuously generates dynamic log files as small files, and is no longer a single large file.

[0073] Step 502, using the system log service tool, reading log data into the latest dynamic log file;

[0074] Since the dynamic log file is generated continuously at a fixed time, the embodiment of the present application utilizes the system log service tool to read the log data into the latest dynamic log file each time a new dynamic log file is generated, and before the next new dynamic log file is generated, the log data is written to the dynamic log file until the next new dynamic log file is generated, and the log data is switched to be read into the latest dynamic log file. The size of the generated dynamic log file varies depending on the amount of log data generated during this period.

[0075] Step 503, calling the log merging script in the user state of the Linux operating system, and when it is determined by executing the log merging script that the log file transfer storage condition is met, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back;

[0076] In this embodiment, a log merging script is newly added in the Linux operating system. The log merging script has the permission to access the dynamic log file. Specifically, the memory directory of the dynamic log file can be pre-configured. Then, whether the log file transfer storage condition is met is determined by calling the log merging script. When the log file transfer storage condition is met, the currently generated dynamic log file can be accessed and its generation sequence can be determined. According to the generation sequence, the older dynamic log files can be merged and the newer dynamic log files can be reserved. The newer dynamic log files can be reserved by reserving at least one of the latest dynamic log files.

[0077] Step 504: transfer the merged log files to the disk file system.

[0078] The embodiment of the present application changes the way the system log service tool rsyslog generates log files, from writing log data only to a single large log file, to continuously and dynamically generating small dynamic log files. Each dynamic log file is used to store log data for a period of time. By adding a merge script file in the user state, when it is determined that the log transfer storage conditions are met, newer dynamic log files are reserved, and the remaining dynamic logs are merged and transferred to the disk file system. Therefore, the problem of rsyslogd and logrotate cooperation in the related technology does not exist, thereby ensuring that log data will not be lost 100%, and ensuring the reliability of the Linux operating system operation.

[0079] In some possible embodiments, at least one dynamic log file is reserved in the order of generation time from the last to the first, and the remaining dynamic log files are merged in the order of time from the first to the last, including:

[0080] The latest dynamic log file is reserved in the order of generation time from the latest to the earliest, and all other dynamic log files are merged in the order of generation time from the earliest to the latest.

[0081] When the conditions for log file transfer storage are met, the embodiment of the present application reserves a most recently generated dynamic log file according to the generation sequence of the dynamic log files. As long as this dynamic log file is guaranteed and all other dynamic logs are merged, it can be ensured that the system log service tool always writes log data to the latest dynamic log file without losing log data due to transfer storage.

[0082] Through the crond daemon process in the Linux operating system for periodically executing user-defined script files, the user-mode log merging script is called once at a set interval. The set interval time can be set to be shorter, so that when the log transfer storage conditions are met, the dynamic log files are merged and transferred. Exemplarily, the script file log merging script can be set to be called once every minute.

[0083] In some possible embodiments, a system log service tool of the Linux operating system is used to periodically generate dynamic log files in the memory file system, including: using the system log service tool of the Linux operating system to determine when a set interval has been reached since the last dynamic log file was generated, and using the current latest dynamic log file as a cut log file; generating a new dynamic log file to store log data that is subsequently read.

[0084] The embodiment of the present application applies the system log service tool Rsyslogd which has the ability to generate small files. By using Rsyslogd to generate dynamic log files, the current latest dynamic log file can be automatically used as a cut log file for log cutting according to the set interval, exemplarily by minutes. This is equivalent to Rsyslogd doing the work of logrotate, so the problem of rsyslogd and logrotate coordination existing in the related technology does not exist.

[0085] In some possible embodiments, when it is determined that the log file transfer storage conditions are met by executing the log merging script, at least one dynamic log file is reserved in order of generation time from back to front, and the remaining dynamic log files are merged in order of time from front to back, including: monitoring each time a cut log file is generated by executing the log merging script, splicing the cut log file with the previous cut log file; when it is determined that the size of the spliced ​​cut log file exceeds the set file threshold, it is determined that the log file transfer storage conditions are met.

[0086] The embodiment of the present application uses a newly added log merging script logcollector to merge dynamic log files according to the generation sequence of dynamic log files and in the order of generation time from back to front, that is, merge the dynamic log files with older time and reserve the newer dynamic log files. When reserving, one or more can be reserved. In order to realize the merging of dynamic log files according to the generation sequence, a possible method of the embodiment of the present application is to dynamically monitor whether a cut log file is generated, that is, a new dynamic log file is generated, and the previous dynamic log file is used as the newly generated cut log file. Then, the log merging script logcollector is used to splice the cut log file with the previous cut log file. Since the cut log file is generated in chronological order, the spliced ​​cut log files in the embodiment of the present application are spliced ​​according to the generation sequence, thereby supporting the merging according to the generation time sequence. In the embodiment of the present application, the log merging script logcollector is also used to determine whether the size of all currently spliced ​​cut log files exceeds the set file threshold when splicing each cut log file. If it exceeds the set file threshold, it is determined that the log file transfer storage condition is met. Of course, the log merging script logcollector can also determine whether the log file transfer conditions are met according to the number of cut log files or the time since the last transfer.

[0087] In some possible embodiments, the initially generated dynamic log file is named with the first file name, and the subsequently generated dynamic log file name includes the first file name plus an extension, wherein the extension carries timing information; the merged log file uses the second file name, and the name of the latest dynamic log file reserved after the merge is updated to the first file name. Exemplarily, the initially generated dynamic log file is named message, and the subsequently generated dynamic log file is named message.1. Message is used as a cut log file. The subsequently generated dynamic log file is named message.2. Message.1 is used as a cut log file and is spliced ​​with message to obtain message and message.1 spliced ​​together. The subsequently generated dynamic log file is named message.3. Message.2 is used as a cut log file and is spliced ​​with message and message.1 to obtain message, message.1, and message.2 spliced ​​together. The subsequently generated dynamic log file is message.4, and if it is determined that the spliced ​​message, message.1, and message.2 exceed the file threshold, message, message.1, and message.2 are merged together and transferred to the disk file system. At this time, the newly generated dynamic log file message.4 is updated to message, and the splicing, transfer and file naming methods of the subsequently generated dynamic log files are the same as above.

[0088] In some possible embodiments, the latest dynamic log file is reserved in order of generation time from back to front, and all other dynamic log files are merged in order of time from front to back, including: reserving the latest dynamic log file in order of generation time from back to front, and determining the generation sequence according to the extensions of the remaining dynamic log files; merging in order from front to back according to the generation sequence, using the second file name for the merged log file, and updating the name of the reserved latest dynamic log file to the first file name.

[0089] In the embodiment of the present application, the extension carries timing information, for example, carrying the timing information 1, 2, 3... etc. generated above. Therefore, when the log conversion storage conditions are met according to the extension, the dynamic log files can be merged according to the extension in order from front to back in time.

[0090] In some possible embodiments, the log file transfer storage conditions are determined by executing the log merging script, including: monitoring the time length from the last dynamic log file merge to reach a set time length, and determining that the log file transfer storage conditions are met; or determining that the log file transfer storage conditions are met based on a comparison result of the size of the currently generated dynamic log file with a set file threshold.

[0091] By monitoring dynamic log files in the above manner, it can be ensured that rsyslogd will transfer the dynamic log files to relatively large log files when they reach a relatively large size, and will not frequently transfer the log files. In addition, the latest dynamic log files in the form of small files will be reserved to ensure that the latest written log data will not be lost.

[0092] The newly added log merging script in the embodiment of the present application includes multiple processes, which may specifically include a process for monitoring whether the conditions for log conversion storage are met, a process for cutting and splicing the previously generated dynamic log files when a dynamic log file is generated, and a process for storing the merged log files to the disk file system when the conditions for log conversion storage are met.

[0093] The following is combined with Figure 6 The detailed process of the log data transfer storage method provided in the embodiment of the present application is described as follows: Figure 6 As shown, the method includes:

[0094] Step 6.1, rsyslogd continuously writes the newly generated log data to the dynamic log file initially generated in the memory, namely the messages file;

[0095] The log transfer storage condition can be to monitor that the time from the last file cutting reaches a set time, or the size of the monitored log file reaches a set file threshold, to determine that the trigger condition is met.

[0096] Step 6.2: rsyslogd generates a dynamic log file messages.1 one minute after the messages file is generated. Therefore, one minute after the messages file is generated, the dynamic log file to which the log data is written is switched to messages.1.

[0097] Step 6.3, the log merging script logcollector uses the messages file as the cut log file and splices the cut dynamic log file;

[0098] Step 6.4, rsyslogd continuously writes log data to messages.1 before the next dynamic log is generated;

[0099] Step 6.5, rsyslogd generates a dynamic log file messages.2 one minute after generating the messages.1 file. Therefore, one minute after generating the messages.1 file, the dynamic log file to which the log data is written is switched to messages.2.

[0100] Step 6.6, the log merging script logcollector uses the messages file as the cut log file and splices the dynamic log file processed by the cut, that is, splices the messages file with the messages.1 file;

[0101] In step 6.7, when the log merging script logcollector determines that the amount of spliced ​​logs reaches the set threshold, it determines that the log transfer storage conditions are met, merges the spliced ​​messages file and the messages.1 file, and transfers them to the disk file system.

[0102] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0103] Based on the same inventive concept, the present application also provides a log data transfer storage device, such as Figure 7 As shown, the device comprises:

[0104] The dynamic log file generation module 701 is used to generate dynamic log files in the memory file system at regular intervals by using the system log service tool of the Linux operating system;

[0105] The log data reading module 702 is used to read the log data into the latest dynamic log file using the system log service tool;

[0106] The log file merging module 703 is used to call the log merging script in the user state of the Linux operating system, and when it is determined by executing the log merging script that the log file transfer storage condition is met, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back;

[0107] The log transfer module 704 is used to transfer the merged log files to the disk file system.

[0108] In some possible embodiments, the log file merging module reserves at least one dynamic log file in the order of generation time from the back to the front, and merges the remaining dynamic log files in the order of time from the front to the back, including:

[0109] The latest dynamic log file is reserved in the order of generation time from the latest to the earliest, and all other dynamic log files are merged in the order of generation time from the earliest to the latest.

[0110] In some possible embodiments, the log file merging module determines that the log file transfer storage condition is satisfied by executing the log merging script, including:

[0111] The monitoring time from the last merging of dynamic log files reaches the set time, and the log file transfer storage conditions are determined to be met; or

[0112] According to the comparison result of the size of the currently generated dynamic log file and the set file threshold, it is determined that the log file transfer storage condition is met.

[0113] In some possible embodiments, the dynamic log file generation module utilizes the system log service tool of the Linux operating system to periodically generate a dynamic log file in the memory file system, including:

[0114] Using the system log service tool of the Linux operating system, determine that when the interval from the last dynamic log file generation reaches a set interval, the latest dynamic log file is used as the cut log file;

[0115] Generate a new dynamic log file to store the log data read subsequently.

[0116] In some possible embodiments, when the log file merging module determines that the log file transfer storage condition is met by executing the log merging script, at least one dynamic log file is reserved in the order of generation time from back to front, and the remaining dynamic log files are merged in the order of time from front to back, including:

[0117] By executing the log merging script, each time a cut log file is generated, the cut log file is spliced ​​with the previous cut log file;

[0118] When it is determined that the size of the spliced ​​cutting log file exceeds the set file threshold, it is determined that the log file transfer storage condition is met.

[0119] In some possible embodiments, the initially generated dynamic log file is named as the first file name, and the subsequently generated dynamic log file name includes the first file name plus an extension, wherein the extension carries timing information;

[0120] The merged log file uses the second file name, and the name of the latest dynamic log file reserved after the merger is updated to the first file name.

[0121] In some possible embodiments, the log file merging module reserves the latest dynamic log file in the order of generation time from the latest to the latest, and merges all other dynamic log files in the order of time from the earliest to the latest, including:

[0122] In order of generation time from the latest to the earliest, the latest dynamic log file is reserved, and the generation sequence of the remaining dynamic log files is determined according to their extensions.

[0123] The log files are merged in a time-to-time order according to the generation sequence, the merged log files are given the second file name, and the name of the reserved latest dynamic log file is updated to the first file name.

[0124] After introducing the log data transfer storage method and apparatus according to the exemplary embodiment of the present application, next, a log data transfer storage device according to another exemplary embodiment of the present application is introduced.

[0125] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0126] In some possible implementations, the log data storage device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the log data storage method according to various exemplary implementations of the present application described above in this specification.

[0127] Refer to the following Figure 8 The log data transfer storage device 180 according to this embodiment of the present application is described. Figure 8 The log data transfer storage device 180 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] like Figure 8 As shown, the log data transfer storage device 180 is in the form of a general electronic device. The components of the log data transfer storage device 180 may include but are not limited to: at least one processor 181, at least one memory 182, and a bus 183 connecting different system components (including the memory 182 and the processor 181).

[0129] Bus 183 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0130] The memory 182 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1821 and / or a cache memory 1822 , and may further include a read-only memory (ROM) 1823 .

[0131] The memory 182 may also include a program / utility 1825 having a set (at least one) of program modules 1824, such program modules 1824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0132] The log data transfer storage device 180 can also communicate with one or more external devices 184 (such as keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable users to interact with the log data transfer storage device 180, and / or communicate with any device (such as routers, modems, etc.) that enables the log data transfer storage device 180 to communicate with one or more other electronic devices. This communication can be carried out through an input / output (I / O) interface 185. In addition, the log data transfer storage device 180 can also communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 186. As shown in the figure, the network adapter 186 communicates with other modules of the log data transfer storage device 180 through a bus 183. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the log data transfer storage device 180, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0133] In some possible implementations, various aspects of a log data transfer and storage method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a log data transfer and storage method according to various exemplary implementations of the present application described above in this specification.

[0134] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] The program product for log data transfer storage of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0136] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0138] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0139] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0140] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and block diagram, as well as the combination of the processes and boxes in the flowchart and block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and block diagram. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A log data transfer storage method, characterized in that: The method comprises: Use the system log service tool of the Linux operating system to generate dynamic log files in the memory file system at regular intervals; Using the system log service tool, read the log data into the latest dynamic log file; Calling a log merging script in the user state of the Linux operating system, and when it is determined by executing the log merging script that the log file transfer storage condition is met, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back; The merged log files are transferred to the disk file system.

2. The method according to claim 1, characterized in that: At least one dynamic log file is reserved in the order of generation time from the latest to the latest, and the remaining dynamic log files are merged in the order of generation time from the earliest to the latest, including: The latest dynamic log file is reserved in the order of generation time from the latest to the earliest, and all other dynamic log files are merged in the order of generation time from the earliest to the latest.

3. The method according to claim 1, characterized in that The log file transfer storage condition is determined to be satisfied by executing the log merging script, including: The monitoring time from the last merging of dynamic log files reaches the set time, and the log file transfer storage conditions are determined to be met; or According to the comparison result of the size of the currently generated dynamic log file and the set file threshold, it is determined that the log file transfer storage condition is met.

4. The method according to claim 1, characterized in that: Use the system log service tool of the Linux operating system to generate dynamic log files in the memory file system at regular intervals, including: Using the system log service tool of the Linux operating system, determine that when the interval from the last dynamic log file generation reaches a set interval, the latest dynamic log file is used as the cut log file; Generate a new dynamic log file to store the log data read subsequently.

5. The method according to claim 4, characterized in that When the log file transfer storage condition is determined to be met by executing the log merging script, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back, including: By executing the log merging script, each time a cut log file is generated, the cut log file is spliced ​​with the previous cut log file; When it is determined that the size of the spliced ​​cutting log file exceeds the set file threshold, it is determined that the log file transfer storage condition is met.

6. The method according to claim 2, characterized in that The initially generated dynamic log file is named as the first file name, and the subsequently generated dynamic log file name includes the first file name plus an extension, wherein the extension carries timing information; The merged log file uses the second file name, and the name of the latest dynamic log file reserved after the merger is updated to the first file name.

7. The method according to claim 6, characterized in that In order of generation time, keep the latest dynamic log file, and merge all other dynamic log files in order of time, including: In order of generation time from the latest to the earliest, the latest dynamic log file is reserved, and the generation sequence of the remaining dynamic log files is determined according to their extensions. The log files are merged in a time-to-time order according to the generation sequence, the merged log files are given the second file name, and the name of the reserved latest dynamic log file is updated to the first file name.

8. A log data transfer storage device, characterized in that: The device comprises: A dynamic log file generation module is used to utilize the system log service tool of the Linux operating system to periodically generate dynamic log files in the memory file system; A log data reading module, used to read log data into the latest dynamic log file using the system log service tool; The log file merging module is used to call the log merging script in the user state of the Linux operating system. When it is determined that the log file transfer storage condition is met by executing the log merging script, at least one dynamic log file is reserved in the order of generation time from the back to the front, and the remaining dynamic log files are merged in the order of time from the front to the back; The log transfer module is used to transfer the merged log files to the disk file system.

9. A log data transfer storage device, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images