Log-to-storage method, device and equipment based on system framework component

Abstract

The invention discloses a log-to-storage method, device and equipment based on a system framework component. The method comprises the following steps: reading log data to a log file by utilizing a system log service tool of a Linux operating system; when monitoring that a triggering condition is reached through the system framework component, taking the current log file as a cut log file, creating a new log file and notifying a system log service tool; calling a script file in a user mode through the daemon process, and determining the current processing progress of the system log service tool and the system framework component by executing the script file; and when it is determined that a log-to-storage condition is met according to the current processing progress of a system log service tool and a system framework component, the cut log file is transferred to a disk file system. According to the method, the problem that log data may be lost due to the fact that log is converted into storage through logate and Rsyslod is used for collecting files into new log files in an asynchronous mode in the prior art is solved.

Description

Background Art

[0002] In the Linux operating system, the system log service tool Rsyslogd is used for log collection and log filtering. The input source is the kernel log ring buffer, and the output destination is the log file, i.e., the messages file. The default configuration file of Rsyslogd is configured in the specified directory. The file name and path of the messages file can be specified or modified.

[0003] Log cutting means that when a log file reaches the set trigger condition, it is cut / split, similar to truncation, and the original log file with a larger capacity is "robbed" and transferred to another log file for archiving. The logs generated after this moment continue to be output to the log file whose file header is reset to 0. This process is also commonly known as log rotation.

[0004] Currently, the Linux operating system uses the encapsulated system framework component logrotate to transfer files when the monitoring reaches the trigger condition. However, since the system framework component logrotate provides users with only configuration file modification and shell script callback interfaces, it cannot find a suitable time to do log dumping. It is an asynchronous operation with the system log service tool Rsyslogd to collect files into new log files. Therefore, log data may be lost, and kernel log data loss will seriously affect the normal operation of the Linux operating system. Summary of the invention

[0005] The purpose of this application is to provide a log storage method, device and equipment based on a system framework component, which is used to solve the problem that the existing log storage using logrotate and Rsyslogd collecting files into new log files are asynchronous operations, and there may be a situation where log data is lost.

[0006] In a first aspect, an embodiment of the present application provides a log transfer storage method based on a system framework component, the method comprising:

[0007] Use the system log service tool of the Linux operating system to read log data into the log file;

[0008] When the trigger condition is reached through the monitoring of the system framework component of the Linux operating system, the current log file is used as the cut log file, a new log file is created and the system log service tool is notified;

[0009] Calling a user-mode script file through a daemon process of a Linux operating system, and determining the current processing progress of a system log service tool and a system framework component by executing the script file;

[0010] When it is determined that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component, the cut log file is transferred and stored to the disk file system.

[0011] The embodiment of the present application adds a user-mode script file in the Linux operating system. When the system log service tool is used to read log data into the log file and the system framework component monitors and reaches the trigger condition for log cutting, the user-mode script file is called by the daemon process of the Linux operating system to determine the current processing progress of the system log service tool and the system framework component, thereby determining whether to execute log file transfer storage based on the progress, and indirectly realizing the handshake between the system framework component and the system log service tool to ensure that the transfer storage is executed when the log data is successfully written to the new log file, thereby ensuring that the log data will not be lost 100%, and ensuring the reliability of the Linux operating system operation.

[0012] In some possible embodiments, determining whether the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component includes:

[0013] If there are currently a cut log file and a new log file, determine that the system framework component completes file cutting, and if the new log file contains log data, determine that the system log service tool successfully reads the log data into the new log file;

[0014] If the system framework component completes file cutting and the system log service tool successfully reads the log data into a new log file, it is determined that the log transfer storage conditions are met.

[0015] In the embodiment of the present application, the newly added script file in the user state can check whether the new log file is empty. If it is empty, it means that rsyslogd has not updated the handle of the log file after receiving the notification, or has not successfully written the log to the new log file after updating the handle. If the new log is not empty, that is, there is log data, it is determined that the system log service tool successfully reads the log data into the new log file.

[0016] In some possible embodiments, calling a user-mode script file through a daemon process of a Linux operating system includes:

[0017] The daemon crond in the Linux operating system, which is used to periodically execute user-defined script files, calls the user-mode script file logtransfer at a set interval. This allows timely detection of whether file cutting has occurred and whether the new log file has been successfully written into the log data. When it is detected that the log dump conditions are met, the log transfer is executed, thereby achieving timing control.

[0018] In some possible embodiments, the method further includes:

[0019] The callback function used to transfer the cut log files to the disk file system, which is called by the system framework component of the Linux operating system through the script callback interface, is set to an empty function in advance. This is equivalent to removing the function of transferring the log files performed by logrotate. The function of transferring the log files is implemented by the newly added user script file, which is highly customizable.

[0020] In some possible embodiments, using the current log file as a cut log file, creating a new log file and notifying the system log service tool includes:

[0021] Add the specified extension to the file name of the current log file to obtain the cut log file; create a new log file with the same name as the log file before the extension is added;

[0022] Make sure that there are currently cut log files and new log files, including:

[0023] If the file name of the currently stored first file carries a specified extension, and the file name of the second file is the same as the file name of the first file after removing the extension, it is determined that a cut log file and a new log file currently exist.

[0024] The embodiment of the present application utilizes the above-mentioned file naming method to ensure that rsyslogd always writes log data to the same named log file, and for a newly added script file, the relationship between the two log file names can be used to search in the memory to see whether there are cut log files and created log files, and it can be distinguished which is the cut log file and which is the created log file.

[0025] In some possible embodiments, monitoring the reaching of a trigger condition through a system framework component of a Linux operating system includes:

[0026] The system framework component of the Linux operating system monitors that the time from the last file cutting reaches the set time, and determines that the trigger condition is met; or

[0027] The system framework component of the Linux operating system monitors the size of the log file to reach a set file threshold, and determines that a trigger condition is met.

[0028] The embodiment of the present application can timely determine whether the file cutting conditions are met through the above two monitoring methods to achieve trigger conditions, thereby realizing timely cutting of log files.

[0029] In some possible embodiments, when a trigger condition is reached through monitoring by a system framework component of a Linux operating system, a new log file is created and a system log service tool is notified, including:

[0030] When the trigger condition is reached through the monitoring of the system framework component of the Linux operating system, a new log file is created and a pending SIGHUP signal is sent to the system log service tool, so that the system log service tool starts to read log data to the new log file when receiving the SIGHUP signal.

[0031] The embodiment of the present application sends a SIGHUP signal to the system log service tool through the system framework component, thereby realizing timely notification of log cutting.

[0032] In a second aspect, an embodiment of the present application provides a log-to-storage device based on a system framework component, the device comprising:

[0033] The log data reading module is used to read the log data into the log file by using the system log service tool of the Linux operating system;

[0034] The log cutting module is used to monitor the system framework component of the Linux operating system and when the trigger condition is reached, the current log file is used as the cutting log file, a new log file is created and the system log service tool is notified;

[0035] A progress determination module is used to call a user-mode script file through a daemon process of a Linux operating system, and determine the current processing progress of a system log service tool and a system framework component by executing the script file;

[0036] The log transfer storage module is used to transfer the cut log file to the disk file system when determining that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component.

[0037] In the third aspect, another embodiment of the present application also provides a log conversion storage device based on the system framework component, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any log conversion storage method based on the system framework component provided in the embodiment of the present application.

[0038] In a fourth aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any log conversion storage method based on a system framework component provided in an embodiment of the present application.

[0039] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 This is a schematic diagram of the process of using rsyslogd to read log data and store it in the messages file in the related technology;

[0042] Figure 2 A schematic diagram of the log cutting process for the related technology;

[0043] Figure 3 A schematic diagram of a log cutting process of a disk file system in the related art;

[0044] Figure 4 A schematic diagram of a log cutting process of a memory file system in a related art;

[0045] Figure 5 A flow chart of a log storage method based on a system framework component provided according to an embodiment of the present application;

[0046] Figure 6 A detailed flow chart of a log transfer storage method based on a system framework component according to an embodiment of the present application;

[0047] Figure 7 This is a structural diagram of a log-to-storage device based on a system framework component according to an embodiment of the present application;

[0048] Figure 8 The present invention is a structural diagram of a log-to-storage device based on a system framework component according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0050] In the Linux operating system, there are two input sources for log data generation. One input source is the kernel log data generated by the kernel prink function, and the other input source is the user-mode log data generated by executing the user-mode logger command through the device interface. The device interface can be dynamically generated during process management, file management, and device management in the Linux operating system. The rsyslogd of the Linux operating system is a user-mode process that is responsible for continuously retrieving the log data from the cache and storing it in the messages file. Figure 1 As shown, the input part includes the above-mentioned input source. The log data generated by the above-mentioned input source is queued after being pre-processed by the system log service tool rsyslogd. The corresponding log write request is parsed by calling the HTTP message parser and responded to. Specifically, the queued log data is filtered by the filter engine and then added to the corresponding action queue action aueue. The log data is output to the messages file through the action queue action aueue through the action processor Action Processor.

[0051] Log cutting means that when the message files reach the set trigger conditions, such as according to a certain time period (daily), or according to the file size (100MB), they are cut / split, similar to truncation, and the original large log file is "robbed" and transferred to another log file for archiving. The logs generated after this moment continue to be output to the new log file whose file header is reset to 0. This process is also commonly known as log rotation. The encapsulated system framework component logrotate currently provided by the Linux operating system can automatically rotate, compress, delete and mail log files according to the configuration file to reduce the size of the log file and maintain the normal operation of the system. Figure 2The example log rolling process is shown. Specifically, after the Linux operating system starts the application, the log file is 0MB at initialization. After one day of running, the log file reaches 500MB, meeting the threshold for cutting the log file. The log file is cut, the existing 500MB log file is transferred and a new 0MB log file is created. The file name of the new log file is the same as the file name of the cut log file.

[0052] After Rsyslogd collects kernel log data and user-mode log data, the storage destination, i.e., the log file, may be in the disk file system or in the memory file system. If rsyslogd reads the generated log data into the log file directly in the disk file system, when the log file splitting condition is met, it only needs to rename the transferred log file and create a new log file with the same file name as the original one, and there is no need for log dumping. That is, logrotate creates a new file, and before rsyslogd receives the notification, it still writes the log to the previous log file, regardless of whether the file name is renamed. Figure 3 The following is a schematic diagram of the log rolling process of the output directory of log data in the disk file system scenario, which mainly includes:

[0053] Step 1: The Linux operating system continuously generates log data while running the APP. rsyslogd stores the generated log data in the disk file system. When Logrotate determines that the conditions for file cutting are met, it starts cutting the log file (mv old log file). Specifically, Logrotate renames the old log file.

[0054] Step 2: Logrotate creates a new empty log file with the same name as the old log file.

[0055] Step 3, Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification;

[0056] Step 4: rsyslogd changes the handle of the log file from pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data is written to the new log file.

[0057] If the log file is in the memory file system, rsyslogd will read the generated log data into the memory log file. Therefore, when the log file cutting conditions are met, the memory log file needs to be transferred to the disk file system and a new log file with the same file name as the original one needs to be created. That is, in the scenario where the log output directory is in the memory file system, you need to add a "log dump". That is, move the renamed log file from the memory file system to the disk file system. Executing the mv command across file systems essentially copies the file first and then deletes the original file. Rsyslogd and logrotate rely on sending a SIGHUP signal to switch the file's logrotate to create a new file. Before rsyslogd receives the notification, it still writes the log to the previous log file, regardless of whether the file name has been transferred. For example Figure 4 The following is a schematic diagram of the log rolling process of the output directory of log data in the memory file system scenario, which mainly includes

[0058] Step 1: The Linux operating system continuously generates log data while running the APP. rsyslogd stores the generated log data in the memory file system. When Logrotate determines that the conditions for file cutting are met, it starts to cut the log file (mv the old log file). Specifically, Logrotate renames the old log file.

[0059] Step 2: Logrotate creates a new empty log file with the same name as the old log file.

[0060] Step 3, Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification;

[0061] In step 4.1, rsyslogd changes the handle of the log file from pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data will be written to the new log file.

[0062] In step 4.2, rsyslogd transfers the renamed log files to the disk file system.

[0063] For log files, the transfer of log files is performed by Logrotate, which is equivalent to the log file reading end, and the output of newly generated log data to log files is completed by rsyslogd, which is equivalent to the log file writing end. However, the writing end and the reading end are not synchronized at present. At present, when Logrotate detects that the log cutting conditions are met, it only sends a suspended SIGHUP signal to rsyslogd, and there is no handshake between the two. Therefore, the above steps 4.1 and 4.2 are performed concurrently, and there is no guarantee that step 4.2 will be executed after step 4.1. In this way, it is possible that before rsyslogd has successfully switched to writing log data to the new log file, Logrotate has already executed the cutting and transfer of log files, so that the log data cannot be correctly written to the log file, resulting in the loss of log data.

[0064] Logrotate is a system framework component encapsulated in the Linux operating system. It can automatically rotate, compress, delete and mail log files according to the configuration file to reduce the size of the log file and maintain the normal operation of the system. The interface provided to the user is only the configuration file modification and shell script callback interface. Through the interface provided to the user, it is impossible to find the time point for switching between the old and new logs, so it is impossible to find a suitable time to do log dump. It is impossible to complete the synchronization of step 4.1 and step 4.2 through the interface provided to the user by Logrotate, that is, the process of Logrotate cutting and transferring log files cannot be realized, and it is strongly coupled with the process of rsyslogd writing log data to the log file.

[0065] In view of the fact that logrotate is used to transfer logs and Rsyslogd collects files into new log files in the related art, which is an asynchronous operation and may cause log data loss, the embodiment of the present application provides a log transfer method based on a system framework component, which is applied to the Linux operating system, such as Figure 5 As shown, the method includes:

[0066] Step 501, using the system log service tool of the Linux operating system to read log data into a log file;

[0067] In the embodiment of the present application, the system log service tool can be, but is not limited to, the system log service tool rsyslog provided by the current Linux operating system. Rsyslog is a powerful system log service tool. The main functions are log collection and log filtering. The specific process of rsyslog writing log data to the log file is: the log data generated by the above-mentioned input source is queued after pre-processing by the system log service tool rsyslogd, and the corresponding log write request is parsed and responded by calling the HTTP message parser parser, specifically filtering the queued log data through the filter engine and adding them to the corresponding action queue action aueue respectively, and outputting the log data to the messages file through the action processor Action Processor through the action queue action aueue.

[0068] There are two input sources for the generation of log data, one is the kernel log data generated by the kernel prink function, and the other is the user-mode log data generated by executing the user-mode logger command through the device interface. After rsyslogd collects the user-mode log data and kernel log data, it writes them to the messages file of the disk file system or the messages file of the memory file system. The transfer and storage of log data refers to the solution of the related technology, that is, when Logrotate determines that the conditions for file cutting are met, it starts to cut the log file (mv the old log file), specifically, Logrotate renames the old log file; Logrotate creates a new empty log file with the same name as the original old log file name; Logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd updates the handle of the log file after receiving the notification; rsyslogd modifies the handle of the log file from originally pointing to the old log file to executing the new log file, so that when the APP generates new log data later, the log data is written to the new log file.

[0069] The embodiment of the present application is mainly used to solve the problem of different cutting, storage and writing of log files corresponding to log data. The system log service tool of the Linux operating system is used to read the log data into the log file, including the system log service tool of the Linux operating system reading the log data into the log file in the memory, that is, the messages file. The messages file storing the log data is configured in the memory directory, and the specific location can be flexibly configured as needed.

[0070] Step 502, when the trigger condition is reached through monitoring by the system framework component of the Linux operating system, the current log file is used as a cut log file, a new log file is created and the system log service tool is notified;

[0071] The system framework component in the embodiment of the present application can be, but is not limited to, logrotate in the current Linux operating system. Logrotate can automatically rotate, compress, delete and mail log files according to the configuration file to reduce the size of the log files and maintain the normal operation of the system. Logrotate includes many processes. When the trigger condition is reached through the corresponding process monitoring, the current log file is used as a cut log file, a new log file is created through another process and the system log service tool is notified.

[0072] Step 503, calling the script file in the user state through the daemon process of the Linux operating system, and determining the current processing progress of the system log service tool and the system framework component by executing the script file;

[0073] The Linux operating system provides a daemon process crond, which is a daemon process in the Linux operating system and is used to periodically execute user-defined commands or scripts.

[0074] The system framework component logrotate only provides users with configuration files and shell script callback interfaces. The location of logrotate can be modified through the configuration file interface. The script file of the system framework component logrotate is a script file written in C language. The script file includes many callback functions. It is necessary to use C language to call shell commands to call the functions. It is relatively troublesome to modify the timing of log rotation storage only through the script callback interface. Therefore, the customizability of the system framework component is poor.

[0075] The embodiment of the present application provides a corresponding script file added in the user state of the Linux operating system. The script file is specifically a script file in the user state called by a daemon process. The current processing progress of the system log service tool and the system framework component is determined by executing the script file. Since the daemon process has the characteristic of periodically executing user-defined commands or scripts, the daemon process can ensure that the newly added script file is periodically executed. The current processing progress of the system log service tool and the system framework component is determined through the script file, and the system log service tool is monitored to determine whether it is currently writing log data to the log file before cutting or writing log data to the newly created log file. It is also monitored whether the system framework has currently completed the log file cutting or is in a state of not cutting the log file.

[0076] Step 504: when it is determined that the log transfer condition is met according to the current processing progress of the system log service tool and the system framework component, the cut log file is transferred to the disk file system.

[0077] In the embodiment of the present application, a new script file is added in the user state, and the daemon process is called regularly. The current processing progress of the system log service tool and the system framework component is determined by the newly added script file. When the log transfer storage condition is met, specifically when the monitoring system log service tool writes log data to the created new log file, and whether the system framework has currently completed the log file cutting, when it is determined that the log transfer storage condition is met, the cut log file is transferred to the disk file system. The handshake between the system framework component and the system log service tool is indirectly realized to ensure that the transfer storage is performed when the log data is successfully written to the new log file, thereby ensuring that the log data will not be lost 100%, and ensuring the reliability of the operation of the Linux operating system.

[0078] In some possible embodiments, determining whether the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component includes:

[0079] If there are currently a cut log file and a new log file, it is determined that the system framework component has completed file cutting. If the new log file contains log data, it is determined that the system log service tool has successfully read the log data into the new log file. In the embodiment of the present application, the newly added script file in the user state can access the log file stored in the memory according to the memory directory of the log file. When logrotate detects that the set trigger condition is reached, it uses the current log file in the memory as the cut log file and creates a new log file in the memory. The newly added script file in the user state can access the log file stored in the memory to determine whether there is only one memory file in the memory, or there are both a cut log file and a new log file. If there are both a cut log file and a new log file, it means that logrotate has completed the cutting of the log file. Further, the newly added script file in the user state can check whether the new log file is empty. If it is empty, it means that rsyslogd has not updated the handle of the log file after receiving the notification, or has not successfully written the log to the new log file after updating the handle. If the new log is not empty, that is, there is log data, it is determined that the system log service tool has successfully read the log data into the new log file.

[0080] If the system framework component completes file cutting and the system log service tool successfully reads the log data into a new log file, it is determined that the log transfer storage conditions are met.

[0081] In some possible embodiments, calling a user-mode script file through a daemon process of a Linux operating system includes:

[0082] The daemon process crond in the Linux operating system, which is used to periodically execute user-defined script files, calls the user-mode script file logtransfer once at a set interval.

[0083] Exemplarily, the script file logtransfer may be set to be called once every minute, so that it is possible to promptly detect whether file cutting has occurred and whether the new log file has successfully written log data, and execute log transfer storage when it is detected that the log transfer conditions are met, thereby achieving timing control.

[0084] In some possible embodiments, the method further includes:

[0085] The callback function for transferring and storing the cut log file to the disk file system, which is called by the system framework component of the Linux operating system through the script callback interface, is set to an empty function in advance.

[0086] As mentioned above, the system framework component logrotate only provides users with configuration files and shell script callback interfaces. The script file includes many callback functions, and it is necessary to use C language to call shell commands to call functions. It is relatively troublesome to modify the timing of log transfer storage only through the script callback interface. The embodiment of the present application can pre-set the callback function used to transfer the cut log file to the disk file system through the script callback interface of the system framework component to an empty function, which is equivalent to removing the function of transferring storage performed by logrotate. The function of log transfer storage is implemented by the newly added user's script file, which is highly customizable.

[0087] In some possible embodiments, the current log file is used as a cut log file, a new log file is created and the system log service tool is notified, including: adding a specified extension to the file name of the current log file to obtain a cut log file; creating a new log file with the same name as the log file before the extension is added. Determining whether there is a cut log file and a new log file currently includes: if the file name of the currently stored first file carries a specified extension, and the file name of the second file is the same as the file name of the first file after the extension is removed, then determining that there is a cut log file and a new log file currently. The embodiment of the present application utilizes the above-mentioned file naming method to ensure that rsyslogd always writes log data to the same named log file, and for the newly added script file, the relationship between the two log file names can be used to retrieve whether there is a cut log file and a created log file in the memory, and it can be distinguished which is the cut log file and which is the created log file.

[0088] In some possible embodiments, the trigger condition is monitored by the system framework component of the Linux operating system, including: monitoring by the system framework component of the Linux operating system that the time from the last file cutting reaches a set time, and determining that the trigger condition is met; or monitoring by the system framework component of the Linux operating system that the size of the log file reaches a set file threshold, and determining that the trigger condition is met.

[0089] Through the above-mentioned monitoring method of log files, it can be ensured that rsyslogd can cut log files in time and no oversized log files will be generated.

[0090] In some possible embodiments, when a trigger condition is reached through monitoring by a system framework component of the Linux operating system, a new log file is created and the system log service tool is notified, including: when a trigger condition is reached through monitoring by a system framework component of the Linux operating system, a new log file is created and a pending SIGHUP signal is sent to the system log service tool, so that the system log service tool starts reading log data to the new log file when receiving the SIGHUP signal.

[0091] The newly added script file execution log transfer storage function in the embodiment of the present application can be implemented by the corresponding process. The process can perform log transfer storage with compression function, and the transferred files will no longer write log data.

[0092] The embodiment of the present application uses a signal mechanism to switch between old and new logs for Logrotate and rsyslogd, but this process is asynchronous. The interface provided by logrotate to the user leads to the problem that the time point of switching between old and new logs cannot be found. The embodiment of the present application adds a user-mode script file, which can customize the dumping conditions and control the conditions for transferring and storing log data, eliminating the shortcoming of poor customizability of logrotate. It ensures that logrotate and rsyslogd no longer modify the dumped files, and can ensure that 100% of the logs are not lost during the log dumping process.

[0093] The following is combined with Figure 6 The detailed process of the log transfer storage method based on the system framework component provided in the embodiment of the present application is described as follows: Figure 6 As shown, the method includes:

[0094] Step 6.1: When rsyslogd determines that the log transfer storage conditions are not met, it continuously writes the newly generated log data to the log file in the memory, namely the messages file;

[0095] The log transfer storage condition can be to monitor that the time from the last file cutting reaches a set time, or the size of the monitored log file reaches a set file threshold, to determine that the trigger condition is met.

[0096] Step 6.2: When logrotate determines that the log transfer storage conditions are met, it re-commands the messages file, specifically by adding an extension;

[0097] Step 6.3, logrotate creates a new log file in the memory file system, namely the new messages file. The new messages file has the same name as the previous messages file.

[0098] Step 6.4, logrotate sends a pending SIGHUP signal to rsyslogd, so that rsyslogd starts reading log data into the new log file when it receives the SIGHUP signal;

[0099] Step 6.5, when rsyslogd receives the SIGHUP signal, it opens a new log file;

[0100] Step 6.6, after rsyslogd opens the new log file, it will subsequently write the log data to the new log file;

[0101] Step 6.7, call the user-mode script file logtransfer once every 1 minute through crond. By executing the logtransfer according to the file name, when it is detected that the messages file and the new messages file exist at the same time, and the new messages file is not empty, it is determined that the log transfer storage conditions are met, and the messages file is transferred to the disk file system.

[0102] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0103] Based on the same inventive concept, the present application also provides a log transfer storage 700 based on a system framework component, such as Figure 7 As shown, the device comprises:

[0104] The log data reading module 701 is used to read the log data into the log file by using the system log service tool of the Linux operating system;

[0105] The log cutting module 702 is used to use the current log file as the cutting log file, create a new log file and notify the system log service tool when the trigger condition is reached through the system framework component monitoring of the Linux operating system;

[0106] The progress determination module 703 is used to call the script file in the user state through the daemon process of the Linux operating system, and determine the current processing progress of the system log service tool and the system framework component by executing the script file;

[0107] The log transfer storage module 704 is used to transfer the cut log file to the disk file system when it is determined that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component.

[0108] In some possible embodiments, the log transfer storage module determines that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component, including:

[0109] If there are currently a cut log file and a new log file, determine that the system framework component completes file cutting, and if the new log file contains log data, determine that the system log service tool successfully reads the log data into the new log file;

[0110] If the system framework component completes file cutting and the system log service tool successfully reads the log data into a new log file, it is determined that the log transfer storage conditions are met.

[0111] In some possible embodiments, the progress determination module calls a user-mode script file through a daemon process of a Linux operating system, including:

[0112] The daemon process crond in the Linux operating system, which is used to periodically execute user-defined script files, calls the user-mode script file logtransfer once at a set interval.

[0113] In some possible embodiments, it further includes:

[0114] The callback function setting module is used to set the callback function called by the system framework component of the Linux operating system through the script callback interface and used to transfer the cut log file to the disk file system to an empty function in advance.

[0115] In some possible embodiments, the log cutting module uses the current log file as the cutting log file, creates a new log file and notifies the system log service tool, including:

[0116] Add the specified extension to the file name of the current log file to obtain the cut log file; create a new log file with the same name as the log file before the extension is added;

[0117] Make sure that there are currently cut log files and new log files, including:

[0118] If the file name of the currently stored first file carries a specified extension, and the file name of the second file is the same as the file name of the first file after removing the extension, it is determined that a cut log file and a new log file currently exist.

[0119] In some possible embodiments, the log cutting module monitors the system framework component of the Linux operating system to reach a trigger condition, including:

[0120] The system framework component of the Linux operating system monitors that the time from the last file cutting reaches the set time, and determines that the trigger condition is met; or

[0121] The system framework component of the Linux operating system monitors the size of the log file to reach a set file threshold, and determines that a trigger condition is met.

[0122] In some possible embodiments, the log cutting module creates a new log file and notifies the system log service tool when a trigger condition is reached through monitoring by the system framework component of the Linux operating system, including:

[0123] When the trigger condition is reached through the monitoring of the system framework component of the Linux operating system, a new log file is created and a pending SIGHUP signal is sent to the system log service tool, so that the system log service tool starts to read log data into the new log file when receiving the SIGHUP signal.

[0124] After introducing the log storage method and apparatus based on the system framework component according to the exemplary embodiment of the present application, next, a log storage device based on the system framework component according to another exemplary embodiment of the present application is introduced.

[0125] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0126] In some possible implementations, the log-to-storage device based on the system framework component according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the log-to-storage method based on the system framework component according to various exemplary implementations of the present application described above in this specification.

[0127] Refer to the following Figure 8To describe the log-to-storage device 180 based on the system framework component according to this embodiment of the present application. Figure 8 The log-to-storage device 180 based on the system framework component is shown as an example only and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] like Figure 8 As shown, the log-to-storage device 180 based on the system framework component is presented in the form of a general electronic device. The components of the log-to-storage device 180 based on the system framework component may include but are not limited to: at least one processor 181, at least one memory 182, and a bus 183 connecting different system components (including the memory 182 and the processor 181).

[0129] Bus 183 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0130] The memory 182 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1821 and / or a cache memory 1822 , and may further include a read-only memory (ROM) 1823 .

[0131] The memory 182 may also include a program / utility 1825 having a set (at least one) of program modules 1824, such program modules 1824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0132] The log-to-storage device 180 based on the system framework component can also communicate with one or more external devices 184 (e.g., keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable users to interact with the log-to-storage device 180 based on the system framework component, and / or communicate with any device (e.g., routers, modems, etc.) that enables the log-to-storage device 180 based on the system framework component to communicate with one or more other electronic devices. Such communication can be performed through an input / output (I / O) interface 185. In addition, the log-to-storage device 180 based on the system framework component can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 186. As shown in the figure, the network adapter 186 communicates with other modules of the log-to-storage device 180 based on the system framework component through a bus 183. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the log-to-storage device 180 based on the system framework component, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] In some possible implementations, various aspects of a log conversion and storage method based on a system framework component provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a log conversion and storage method based on a system framework component according to various exemplary implementations of the present application described above in this specification.

[0134] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] The program product for log transfer storage based on the system framework component of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0136] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0138] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0139] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0140] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and block diagram, as well as the combination of the processes and boxes in the flowchart and block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and block diagram. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A log transfer storage method based on a system framework component, characterized in that: The method comprises: Use the system log service tool of the Linux operating system to read log data into the log file; When the trigger condition is reached through the monitoring of the system framework component of the Linux operating system, the current log file is used as the cut log file, a new log file is created and the system log service tool is notified; Calling a user-mode script file through a daemon process of a Linux operating system, and determining the current processing progress of a system log service tool and a system framework component by executing the script file; When it is determined that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component, the cut log file is transferred and stored to the disk file system.

2. The method according to claim 1, characterized in that Determine whether the log transfer storage conditions are met based on the current processing progress of the system log service tool and system framework components, including: If there are currently a cut log file and a new log file, determine that the system framework component completes file cutting, and if the new log file contains log data, determine that the system log service tool successfully reads the log data into the new log file; If the system framework component completes file cutting and the system log service tool successfully reads the log data into a new log file, it is determined that the log transfer storage conditions are met.

3. The method according to claim 1, characterized in that Calling user-mode script files through the Linux operating system daemon process includes: The daemon process crond in the Linux operating system, which is used to periodically execute user-defined script files, calls the user-mode script file logtransfer once at a set interval.

4. The method according to claim 1, characterized in that Also includes: The callback function for transferring and storing the cut log file to the disk file system, which is called by the system framework component of the Linux operating system through the script callback interface, is set to an empty function in advance.

5. The method according to claim 2, characterized in that: Use the current log file as the cut log file, create a new log file and notify the system log service tool, including: Add the specified extension to the file name of the current log file to obtain the cut log file; create a new log file with the same name as the log file before the extension is added; Make sure that there are currently cut log files and new log files, including: If the file name of the currently stored first file carries a specified extension, and the file name of the second file is the same as the file name of the first file after removing the extension, it is determined that a cut log file and a new log file currently exist.

6. The method according to claim 1, characterized in that The trigger conditions are met through the monitoring of the system framework components of the Linux operating system, including: The system framework component of the Linux operating system monitors that the time from the last file cutting reaches the set time, and determines that the trigger condition is met; or The system framework component of the Linux operating system monitors the size of the log file to reach a set file threshold, and determines that a trigger condition is met.

7. The method according to claim 1, characterized in that When the trigger condition is met through the monitoring of the system framework component of the Linux operating system, a new log file is created and the system log service tool is notified, including: When the trigger condition is reached through the monitoring of the system framework component of the Linux operating system, a new log file is created and a pending SIGHUP signal is sent to the system log service tool, so that the system log service tool starts to read log data to the new log file when receiving the SIGHUP signal.

8. A log storage device based on a system framework component, characterized in that: The device comprises: The log data reading module is used to read the log data into the log file by using the system log service tool of the Linux operating system; The log cutting module is used to monitor the system framework component of the Linux operating system and when the trigger condition is reached, the current log file is used as the cutting log file, a new log file is created and the system log service tool is notified; A progress determination module is used to call a user-mode script file through a daemon process of a Linux operating system, and determine the current processing progress of a system log service tool and a system framework component by executing the script file; The log transfer storage module is used to transfer the cut log file to the disk file system when determining that the log transfer storage condition is met according to the current processing progress of the system log service tool and the system framework component.

9. A log transfer storage device based on a system framework component, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images