Method and device for collecting log round-robin file of linux system

Abstract

The invention relates to a method and a device for collecting log round-robin files of a linux system. The method comprises the following steps: configuring a file path of a log rotation file to be collected in a linux system; when a log collection program is started, obtaining a target log file and / or a target directory based on the file path; generating and storing file state information of the target log file; adding event monitoring to a target directory corresponding to each file path and a parent directory of the target directory; and when the log file meets a log rotation condition, performing log rotation file collection through the event monitoring and the file state information. According to the method and the device for collecting the log rotation file of the linux system, a create mode and a copyruncate mode which are commonly used by a log tool can be supported at the same time, the log file after rotation is accurately recognized, and repeated collection or missing collection caused by a configuration problem is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing, and in particular to a method and device for collecting Linux system log rotation files. Background Art

[0002] In Linux systems, log rotation is often used to prevent log files from growing indefinitely, occupying excessive disk space, and to improve log manageability. Rotation is typically triggered by conditions such as log size or generation time, splitting log files and retaining historical logs. Common log rotation tools in Linux systems, such as logrotate, support two typical rotation modes: create mode, which renames the original log file (such as access.log) to access.log.1 and creates a new access.log file. This requires notifying the writing process to reopen the log file. Copytruncate mode, in which the original log file is copied to access.log.1 and its contents are cleared. The process can continue writing to the same file handle. This mode is suitable for scenarios where the writing process cannot be interrupted.

[0003] Existing log collection systems often configure collection files using preset paths and detect file changes using polling or inotify-based event monitoring. To avoid duplicate collection, file status information is typically generated for each collected file, including file size, modification time, collection offset, and fingerprint information, and stored in memory or a status file. When a new file appears, the collection program determines whether it has already been collected based on the file path or file identifier (such as dev+inode or hash fingerprint), thereby determining the collection starting point.

[0004] While this approach is relatively stable in create mode, it presents numerous issues in copytruncate mode. Because the new files generated by a rotation differ from the original files' inodes, they are often mistakenly identified as new files and collected repeatedly, or log data not yet collected during the rotation period is missed. Furthermore, in copytruncate mode, log content is cleared, but the file handle remains unchanged. Methods that rely on inode comparisons cannot accurately identify rotations, compromising both collection accuracy and robustness.

[0005] On the other hand, the monitoring method based on inotify relies on adding a handle when the file exists, so it cannot preset monitoring when the file has not yet been generated. The detection of new files still requires polling and supplementation, which increases the burden on system resources and has poor real-time performance.

[0006] In summary, the existing technology has problems such as insufficient accuracy, cumbersome configuration, and high error-proneness in log rotation file identification, especially support for copytruncate mode. Therefore, a new method and device for collecting Linux system log rotation files is needed.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0008] In view of this, the present application provides a method and device for collecting Linux system log rotation files, which can simultaneously support the create mode and copytruncate mode commonly used by the logrotate tool, accurately identify the rotated log files, and avoid repeated collection or missed collection due to configuration problems.

[0009] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0010] According to one aspect of the present application, a method for collecting log rotation files of a Linux system is proposed, the method comprising: configuring a file path of a log rotation file to be collected in a Linux system; obtaining a target log file and / or target directory based on the file path when a log collection program is started; generating and saving file status information of the target log file; adding an event listener for each target directory and its parent directory corresponding to the file path; and collecting the log rotation file through the event listener and the file status information when the log file meets the log rotation condition.

[0011] In an exemplary embodiment of the present application, configuring the file path of the log rotation file to be collected in the Linux system includes: configuring the file name and / or directory of the log rotation file to be collected in the Linux system; and configuring the number of bytes corresponding to the file fingerprint.

[0012] In an exemplary embodiment of the present application, when the log collection program is started, the target log file and / or directory is obtained based on the file path, including: when the log collection program is started, obtaining the configured file path; obtaining the target log file and / or target directory based on the file path; deduplicating and summarizing the target log file and / or the target directory.

[0013] In an exemplary embodiment of the present application, generating and saving the file status information of the target log file includes: obtaining basic information of the target log file; calculating fingerprint information of the target log file based on a hash algorithm; generating the file status information through the basic information and the fingerprint information; and saving the file status information into a memory.

[0014] In an exemplary embodiment of the present application, the file status information is generated through the basic information and the fingerprint information, including: obtaining the number of bytes corresponding to the file fingerprint; collecting the offset; generating the file status information through the basic information, the fingerprint information, and the number of bytes corresponding to the file fingerprint; and collecting the offset.

[0015] In an exemplary embodiment of the present application, saving the file status information into memory includes: saving the file status information into a map structure with the file path as the key and the file status information as the value; saving the file status information into a status file with the inode number as the file name.

[0016] In an exemplary embodiment of the present application, event listeners are added to the target directory and its parent directory corresponding to each file path, including: adding an IN_MODIFY event listener and an IN_DELETE_SELF event listener to the target directory corresponding to each file path; adding an IN_MOVED_TO event listener and an IN_CREATE event listener to the parent directory corresponding to each file path and its configured directory path; and adding an IN_DELETE event listener to the configured target directory.

[0017] In an exemplary embodiment of the present application, when a log file meets the log rotation conditions, log rotation file collection is performed through the event monitoring and the file status information, including: setting the log rotation conditions and rotation mode, the rotation mode including the create rotation mode and the copytruncate rotation mode; the log rotation program and the log collection program run in parallel; log rotation file collection is performed through the event monitoring and the file status information.

[0018] In an exemplary embodiment of the present application, performing log rotation file collection through the event monitoring and the file status information includes: identifying the log file to be saved through the event monitoring and the file status information and saving the log file.

[0019] According to one aspect of the present application, a device for collecting log rotation files of a Linux system is proposed, the device comprising: a configuration module for configuring the file path of the log rotation file to be collected in the Linux system; a startup module for obtaining the target log file and / or target directory based on the file path when the log collection program is started; a saving module for generating and saving file status information of the target log file; a monitoring module for adding event monitoring for each target directory and its parent directory corresponding to the file path; and a collection module for collecting log rotation files through the event monitoring and the file status information when the log file meets the log rotation condition.

[0020] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0022] According to the method and device for collecting log rotation files of the Linux system of the present application, the file path of the log rotation file to be collected is configured in the Linux system; when the log collection program is started, the target log file and / or target directory is obtained based on the file path; the file status information of the target log file is generated and saved; and an event monitor is added for each target directory and its parent directory corresponding to the file path; when the log file meets the log rotation conditions, the log rotation file is collected through the event monitor and the file status information. This method can simultaneously support the create mode and copytruncate mode commonly used by the logrotate tool, accurately identify the rotated log file, and avoid repeated collection or missed collection due to configuration problems.

[0023] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features, and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings. The drawings described below are merely some embodiments of the present application, and it is apparent to those skilled in the art that other drawings can be derived from these drawings without inventive effort.

[0025] Figure 1The present invention is a flowchart showing a method for collecting log rotation files of a Linux system according to an exemplary embodiment.

[0026] Figure 2 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0027] Figure 3 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0028] Figure 4 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0029] Figure 5 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0030] Figure 6 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0031] Figure 7 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0032] Figure 8 The figure is a schematic diagram showing a method for collecting log rotation files of a Linux system according to another exemplary embodiment.

[0033] Figure 9 The present invention is a block diagram showing a device for collecting log rotation files of a Linux system according to an exemplary embodiment.

[0034] Figure 10 It is a block diagram of an electronic device according to an exemplary embodiment.

[0035] Figure 11 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0040] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below could be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0041] Those skilled in the art will understand that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0042] The technical abbreviations involved in this application are explained as follows:

[0043] Inotify: A mechanism provided by the Linux kernel for monitoring file system events. It allows programs to monitor changes in files and directories, such as file creation (IN_CREATE), deletion (IN_DELETE), modification (IN_MODIFY), and move-to (IN_MOVED_TO). Inotify works through an event notification mechanism. This means that when a file or directory changes, the relevant event (including the event type and the file path that triggered the event) is immediately notified to the application, minimizing latency and avoiding unnecessary resource consumption associated with polling.

[0044] devInode: In the Linux operating system, an inode is a data structure used to store file metadata in the file system. Each file has a unique inode number within the file system during its lifetime. dev refers to the device number, and inode numbers may be duplicated on different devices. devInode concatenates the two. The dev+inode combination uniquely identifies a file at a given moment. However, since inode numbers are recycled and reused after a file is deleted, the dev+inode combination cannot fully identify a file.

[0045] Hash: A hash algorithm is a function that converts data of arbitrary length into a fixed-length string (often called a hash value). An ideal hash algorithm has a very low probability of producing the same hash value for different inputs (i.e., collisions), so hash values ​​can often be used to uniquely identify the original data. Common hash algorithms include MD5 and SHA256. In the log collection field, the current mainstream method is to perform operations (usually using a hash algorithm) on the first few bytes of a file to obtain a file fingerprint to uniquely identify the file.

[0046] The method proposed in this application aims to simultaneously support the create mode and copytruncate mode commonly used by the logrotate tool, accurately identify rotated log files, and avoid duplicate or missed collection due to configuration problems.

[0047] The technical content of this application is described in detail below with the help of specific embodiments.

[0048] Figure 1 The flowchart of a method for collecting Linux system log rotation files according to an exemplary embodiment is shown. The method 10 for collecting Linux system log rotation files includes at least steps S102 to S110.

[0049] like Figure 1 As shown, in S102, the file path of the log rotation file to be collected is configured in the Linux system. For example, the file name and / or directory of the log rotation file to be collected can be configured in the Linux system; and the number of bytes corresponding to the file fingerprint is configured.

[0050] More specifically, you can configure the file path to be collected in the configuration file. This path can be a directory or a specific file. If a directory is specified, all files within that directory will be collected. Wildcards can be used in the path, for example, / usr / local / ** / *.log will collect all existing and future files or directories matching this pattern. Multiple paths can be configured. Secondly, configure the number of bytes required to calculate the file fingerprint (fingerprint_bytes).

[0051] In S104, when the log collection program is started, the target log file and / or target directory is obtained based on the file path. For example, when the log collection program is started, the configured file path is obtained; the target log file and / or target directory is obtained based on the file path; and the target log file and / or target directory are deduplicated and aggregated.

[0052] More specifically, when the log collection program starts, it reads all configured file paths, finds all existing files and directories that match the configured pattern, aggregates and deduplicates them, and removes duplicate directories and files. If the configured file path is contained in the configured directory, the file path is ignored.

[0053] In S106, file status information of the target log file is generated and saved. For example, basic information of the target log file is obtained; fingerprint information of the target log file is calculated based on a hash algorithm; the file status information is generated using the basic information and the fingerprint information; and the file status information is saved in a memory.

[0054] In one embodiment, generating the file status information from the basic information and the fingerprint information includes: obtaining the number of bytes corresponding to the file fingerprint; collecting an offset; and generating the file status information from the basic information, the fingerprint information, and the number of bytes corresponding to the file fingerprint; and collecting an offset. More specifically, for each file (including all files in the configured directory), basic information including the device number, inode number, file path, file size (file_size), and modification time (modify_time) is obtained, followed by obtaining the first MIN(file_size, fingerprint_bytes) bytes, and calculating the file fingerprint using a HASH algorithm (such as MD5 or SHA256).

[0055] In one embodiment, saving the file status information into memory includes: saving the file status information into a map structure with a file path as a key and the file status information as a value; and saving the file status information into a status file with an inode number as a file name.

[0056] The file status information, including basic file information, file fingerprint (fingerprint), the number of bytes used to calculate the fingerprint (fingerprint_real_bytes), and the collection offset (offset, when first started, you can start collecting from the file header or ignore the previous log and start collecting from the file tail. Here, collection starts from the file header, that is, offset is initialized to 0) is saved to a map structure (state_map) in memory with the file path as the key and the file status information as the value. It is also saved to a status file with the inode number as the file name.

[0057] In S108, event listeners are added for each target directory and its parent directory corresponding to the file path. For example, an IN_MODIFY event listener and an IN_DELETE_SELF event listener are added for each target directory corresponding to the file path; an IN_MOVED_TO event listener and an IN_CREATE event listener are added for each parent directory corresponding to the file path and its configured directory path; and an IN_DELETE event listener is added for the configured target directory.

[0058] like Figure 2 As shown, for each file path (excluding files in the directory), add monitoring of the following inotify events:

[0059] IN_MODIFY event:

[0060] According to the file path, obtain its current file information, and according to its devInode, find the previously recorded status information.

[0061] If not found, it means that the IN_CREATE event of the file has not been processed yet and is temporarily ignored.

[0062] If found, first compare the current file size (cur_file_size) with the previously recorded file size (file_size). If cur_file_size > file_size, then get the previously recorded fingerprint_real_bytes, extract the corresponding part of the current file content, calculate the current file fingerprint (cur_fingerprint), and compare it with the previously recorded fingerprint:

[0063] If cur_file_size > file_size and the file fingerprints match, data collection continues from the previously recorded offset and updates the file status. It also checks whether the previously recorded fingerprint_real_bytes matches the configured fingerprint_bytes. If not, it continues to obtain the first MIN(file_size, fingerprint_bytes) bytes of the file to calculate and update the fingerprint. (The fewer bytes used to calculate the fingerprint, the greater the probability of falsely assuming the files are consistent. In the worst case, logs between offset and fingerprint_bytes may be repeatedly collected.)

[0064] If cur_file_size <= file_size or the file fingerprint is inconsistent, the log file is considered truncated. Check whether the created field exists and whether the field value is true. If so, reset the offset to 0, update the fingerprint, and then start collecting from the beginning; otherwise, ignore it.

[0065] IN_DELETE_SELF event (this event is triggered when the monitored file itself is deleted):

[0066] Delete the recorded status information, and inotify will automatically remove the monitoring of the file. (Note that although the file is deleted, as long as an application keeps the file open, the file will not be recycled by the file system and the application can still read the file. That is, the collection program should keep the file open until the file collection is completed, otherwise some logs may be lost).

[0067] For each file path's parent directory and all configured directory paths, add monitoring for the following inotify events:

[0068] IN_MOVED_TO event (It should be noted that in Linux, after executing mv / path / a / path / b, two events will be triggered in the path directory: IN_MOVED_FROM / path / a and IN_MOVED_TO / path / b; the mv operation does not change the inode number of the file, so if / path / a was previously monitored, it can still be monitored after mv):

[0069] According to the file path, obtain its current file information, and according to its devInode, find the previously recorded status information

[0070] If found, modify and save the recorded file path (state file and state_map), and collect logs from the previously recorded offset to the end of the current file. If the file path is not included in the configured file path, remove the inotify monitoring of the file.

[0071] If not found (this happens when moving from outside the monitoring range to within the monitoring range), compare whether the file path is within the configured file path or the configured directory path. If so, initialize and save its status information and start collecting from the beginning; if not, ignore it.

[0072] IN_CREATE event:

[0073] Taking access.log as an example, there are usually two situations in which this event is triggered:

[0074] 1. In create mode, access.log is renamed to access.log.1, and then access.log is recreated, triggering the IN_CREATE event of access.log

[0075] 2. In copytruncate mode, access.log is copied to access.log1, triggering the IN_CREATE event of access.log.1

[0076] Generally speaking, the file after rotation has the same file name prefix as the file before rotation (the logrotate tool can use some means to change the file name prefix after rotation, but this is not involved in most scenarios). Therefore, the rotation mode can be initially determined by matching the file path prefix. Further determination can be made by using file fingerprints. Specifically:

[0077] Get the file path path in the IN_CREATE event. For each key in state_map, determine whether the expression key*==path is true:

[0078] If it is established, obtain the file status information corresponding to the key (including the file fingerprint fingerprint, the number of bytes used to calculate the file fingerprint fingerprint_real_bytes, the collection offset offset, etc.), and calculate the file fingerprint cur_fingerprint of the file corresponding to the path based on the fingerprint_real_bytes, and then compare it with the fingerprint. If the two are consistent, it means that the file represented by the path is a log file rotated in the copytruncate mode, so that it inherits the offset corresponding to the key, and collects the log with the offset to the end of the current file, and then obtains the file information corresponding to the path, and saves it together with the offset to the state_map and status file. If the configured file path contains path, and the path is not in the configured directory path, you also need to add inotify monitoring for the file. Finally, add the field "created": true to the entry corresponding to the key, indicating that the created event of the rotated file in the copytruncate mode has been processed;

[0079] If all keys in state_map are traversed and no key is found that satisfies key*==path and has the same file fingerprint, the file represented by path is considered to be a new log file that has not been rotated. The system then determines whether the file path matches the configured file path pattern or directory. If so, the system initializes and saves its state information to state_map and the state file, and restarts the collection from the beginning.

[0080] If the directory is a configured file directory rather than a parent directory of a file, you also need to monitor the IN_MODIFY and IN_DELETE events. Their behavior is consistent with the above IN_MODIFY and IN_DELETE_SELF events.

[0081] In S110, when a log file meets a log rotation condition, log rotation files are collected using the event monitoring and the file status information. For example, log rotation conditions and rotation modes can be set, including a create rotation mode and a copytruncate rotation mode; a log rotation program and a log collection program run in parallel; and log rotation files are collected using the event monitoring and the file status information.

[0082] More specifically, the log file to be saved may be identified and saved through the event monitoring and the file status information.

[0083] It should be noted that when the acquisition program is started, the file or directory corresponding to the configured file path may not yet exist, and it is necessary to monitor its generation in some way. You can use inotify to monitor the parent directory or poll the file list. After the file or directory to be monitored is generated, it can be processed according to the above process. The specific implementation details are not explained here because they do not affect the core technical solution of the present invention. In addition, the specific method of log collection and the persistence time of status information are not within the scope of discussion of this application.

[0084] According to the method for collecting log rotation files of a Linux system of the present application, the file path of the log rotation file to be collected is configured in the Linux system; when the log collection program is started, the target log file and / or target directory is obtained based on the file path; file status information of the target log file is generated and saved; event monitoring is added for each target directory and its parent directory corresponding to the file path; when the log file meets the log rotation conditions, the log rotation file is collected through the event monitoring and the file status information. This method can simultaneously support the create mode and copytruncate mode commonly used by the logrotate tool, accurately identify the rotated log file, and avoid repeated collection or missed collection due to configuration problems.

[0085] It should be clearly understood that this application describes how to form and use specific examples, but the principles of this application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in this application, these principles can be applied to many other embodiments.

[0086] The following is a specific example to illustrate the operation process of this method:

[0087] Assume that there is a directory structure in Linux such as Figure 3 As shown, the configuration file of the log collection program is:

[0088] path= / path / dirb / *.log, / path / dirb, / path / dira / a.log

[0089] fingerprint_bytes=512

[0090] When the log collection program starts, the file path obtained after reading the path and removing duplicates is / path / dirb, / path / dira / a.log,

[0091] The initialized state_map is as follows Figure 4 As shown, the status file name and content are as follows Figure 5 As shown. The corresponding relationship of inotify monitoring is as follows Figure 6As shown,

[0092] Assume that / path / dira uses the create rotation mode and / path / dirb uses the copytruncate rotation mode. The rotation conditions for both files are file_size >= 1088. After the rotation, the file suffix uses the default numeric format.

[0093] Now, a 1KB log file is written to / path / dira / a.log. This will first trigger the IN_MODIFY event for / path / dira / a.log. Then, because the file size reaches the log rotation condition, log rotation is triggered. This is followed by the IN_MOVED_TO event for / path / dira / a.log.1 and the IN_CREATE event for / path / dira / a.log. Because the log rotation program and the log collection program run concurrently, two possible scenarios may occur during event processing:

[0094] The first case: When processing the IN_MODIFY event, / path / dira / a.log has not been renamed yet. According to its devInode: 1-1, the corresponding status file is found to exist, and then the data of the corresponding entry in the state_map is obtained through the file path. Then it is found that the previously stored fingerprint_real_bytes is 64, so the first 64 bytes of / path / dira / a.log are extracted to calculate cur_fingerprint. After comparing it with fingerprint, it is found that the two are the same, so the log is collected starting from the previously recorded offset = 0, and the offset is updated to 64 + 1024. At the same time, it is found that fingerprint_real_bytes < fingerprint_bytes, so the first fingerprint_bytes bytes of / path / dira / a.log are extracted to recalculate and update fingerprint; then the IN_MOVED_TO event of / path / dira / a.log.1 is triggered. Since the rename does not change the inode, its devInode is still 1-1. Looking up the status file according to devInode, it is found that the previous file name is / path / dira / a.log, so the key in the corresponding entry in the state_map and the file name stored in the status file are modified, and the collection starts from the offset = 1088 position. Comparing / path / dira / a.log.1 with the configured file path, it is found that it is not in it, so the inotify monitoring of this file is removed; immediately afterwards, the IN_CREATE event of / path / dira / a.log is triggered. According to its file name pattern, no entry satisfying key* == / path / dira / a.log is found in the state_map, so it is considered a new log file, and its status information is obtained and saved.

[0095] After the event processing is completed, the state_map and the status file are as Figure 7 shown (the file information in dirb is omitted).

[0096] Case 2: When processing the IN_MODIFY event, / path / dira / a.log has already been renamed. Regardless of whether / path / dira / a.log has been created, its status file cannot be found based on its devInode. Therefore, the IN_MODIFY event is handled without any action. Next, the IN_MOVED_TO event for / path / dira / a.log.1 is processed. The process is the same as in the first case, except that the collection offset starts at the previously recorded position 0. After the collection is complete, the offset is modified to 1088. The IN_CREATE event is handled in the same way as in the first case.

[0097] After the event processing is completed, except that the fingerprint of / path / dira / a.log.1 is not updated, the rest are the same as above. Figure 1 To.

[0098] Next, let's analyze the execution flow in copytruncate mode. We now write 2KB of log data to / path / dirb / b1.log twice, 1KB each time. There are two situations that affect the processing flow:

[0099] In the first case, when the IN_MODIFY event for / path / dirb / b1.log is processed, / path / dirb / b1.log has not yet been truncated. The IN_MODIFY event processing process is the same as in the first case above and is not detailed here. Subsequently, because the file size reaches the log rotation condition, log rotation is triggered, which in turn triggers the IN_CREATE event for / path / dirb / b1.log.1. When traversing the state_map keys, it finds that / path / dirb / b1.log* == / path / dirb / b1.log.1. Therefore, the first 1024 bytes of / path / dirb / b1.log.1 are used to calculate cur_fingerprint. Finding that this matches the fingerprint of / path / dirb / b1.log, it inherits offset = 2048 of / path / dirb / b1.log. A corresponding entry is added to the state_map, and its state file is initialized. The "created": true field is added to the value of the / path / dirb / b1.log entry in the state_map. This triggers the IN_MODIFY event for / path / dirb / b1.log. Because copy and truncate operations are not atomic, if / path / dirb / b1.log has already been truncated after the data is written but before the IN_MODIFY event is processed, this portion of the log will be lost (this is a flaw in the copytruncate mode itself). When processing the IN_MODIFY event, the current / path / dirb / b1.log file size is obtained. If it is 0 and smaller than the previously stored file size, it is considered to be truncated. The created field in the previously stored file status is checked. If it is true, the offset is reset to 0, the fingerprint is updated, and the created field is removed. After the event is processed, the state_map and the status file are as follows: Figure 8 As shown (files in dira are omitted).

[0100] In the second case, when processing the IN_MODIFY event for / path / dirb / b1.log, / path / dirb / b1.log has already been truncated. Upon discovering the truncation based on the file size, the created field in the previously stored file status is checked. This field is absent, indicating that the IN_CREATE event for the rotated file has not yet been processed. This event is temporarily ignored. The IN_CREATE event processing flow is identical to the first case, except that the offset inherited from / path / dirb / b1.log is set to 0. When processing the IN_MODIFY event for / path / dirb / b1.log again, the truncation is discovered based on the file size. The created field in the previously stored file status is checked and found to be true. The current file information is then updated to the state_map and the status file, with the offset reset to 0 and the created field removed.

[0101] Analysis of the preceding execution flow demonstrates that this method correctly identifies log files before and after rotation in both create and copytruncate modes. Whether the configuration file path includes the path to the rotated file only affects the collection of newly added log files after the rotation; it does not cause duplicate or missed collection of log files before the rotation.

[0102] Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments can be implemented as a computer program executed by a CPU. When executed by the CPU, the computer program performs the functions defined in the above method provided herein. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0103] Furthermore, it should be noted that the aforementioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the aforementioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0104] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0105] Figure 9 This is a block diagram of a device for collecting Linux system log rotation files according to an exemplary embodiment. Figure 9As shown, the device 90 for collecting Linux system log rotation files includes: a configuration module 902 , a startup module 904 , a storage module 906 , a monitoring module 908 , and a collection module 910 .

[0106] The configuration module 902 is used to configure the file path of the log rotation file to be collected in the Linux system;

[0107] The startup module 904 is used to obtain the target log file and / or target directory based on the file path when the log collection program is started;

[0108] The saving module 906 is used to generate and save the file status information of the target log file;

[0109] The monitoring module 908 is used to add event monitoring for each target directory and its parent directory corresponding to the file path;

[0110] The collection module 910 is used to collect log rotation files through the event monitoring and the file status information when the log file meets the log rotation condition.

[0111] According to the device for collecting log rotation files of the Linux system of the present application, the file path of the log rotation file to be collected is configured in the Linux system; when the log collection program is started, the target log file and / or target directory is obtained based on the file path; the file status information of the target log file is generated and saved; and an event monitor is added for each target directory and its parent directory corresponding to the file path; when the log file meets the log rotation conditions, the log rotation file is collected through the event monitor and the file status information. The method can simultaneously support the create mode and copytruncate mode commonly used by the logrotate tool, accurately identify the rotated log file, and avoid repeated collection or missed collection due to configuration problems.

[0112] Figure 10 It is a block diagram of an electronic device according to an exemplary embodiment.

[0113] Refer to the following Figure 10 1000 according to this embodiment of the present application will be described. Figure 10 The electronic device 1000 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0114] like Figure 10As shown, electronic device 1000 is implemented as a general-purpose computing device. Components of electronic device 1000 may include, but are not limited to, at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting various system components (including storage unit 1020 and processing unit 1010), a display unit 1040, and the like.

[0115] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 1010 can perform the following steps: Figure 1 Follow the steps shown in .

[0116] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 10201 and / or a cache memory unit 10202 , and may further include a read-only memory unit (ROM) 10203 .

[0117] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0118] Bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0119] The electronic device 1000 can also communicate with one or more external devices 1000′ (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), devices that allow a user to interact with the electronic device 1000, and / or any device that allows the electronic device 1000 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). This communication can occur via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1060. The network adapter 1060 can communicate with other modules of the electronic device 1000 via the bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 1000, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0120] Through the above description of the embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Figure 11 As shown, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present application.

[0121] The software product can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0122] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0123] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the following functions: configuring a file path for a log rotation file to be collected in a Linux system; obtaining a target log file and / or target directory based on the file path when the log collection program is started; generating and saving file status information of the target log file; adding an event listener for each target directory and its parent directory corresponding to the file path; and collecting the log rotation file through the event listener and the file status information when the log file meets the log rotation condition.

[0125] Those skilled in the art will appreciate that the modules described above can be distributed in the device according to the description of the embodiment, or can be modified accordingly to be used in one or more devices that are different from the embodiment. The modules of the above embodiment can be combined into one module or further divided into multiple submodules.

[0126] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0127] While the exemplary embodiments of the present application have been specifically illustrated and described above, it should be understood that the present application is not limited to the detailed structures, configurations, or implementations described herein; rather, the present application is intended to encompass various modifications and equivalent configurations within the spirit and scope of the appended claims.

Claims

1. A method for collecting Linux system log rotation files, characterized in that: include: Configure the file path of the log rotation file to be collected in the Linux system; When the log collection program is started, the target log file and / or target directory is obtained based on the file path; Generate and save file status information of the target log file; Add event listeners for each target directory and its parent directory corresponding to the file path; When a log file meets the log rotation condition, log rotation file collection is performed through the event monitoring and the file status information.

2. The method according to claim 1, wherein Configure the file path of the log rotation file to be collected in the Linux system, including: Configure the file name and / or directory of the log rotation file to be collected in the Linux system; The number of bytes corresponding to the configuration file fingerprint.

3. The method according to claim 1, wherein When the log collection program is started, the target log file and / or directory is obtained based on the file path, including: When the log collection program is started, the configured file path is obtained; Obtain a target log file and / or target directory based on the file path; Deduplication and aggregation are performed on the target log file and / or the target directory.

4. The method according to claim 1, wherein Generating and saving file status information of the target log file, including: Obtaining basic information of the target log file; Calculate the fingerprint information of the target log file based on a hash algorithm; generating the file status information by using the basic information and the fingerprint information; The file status information is saved in memory.

5. The method according to claim 4, wherein Generating the file status information using the basic information and the fingerprint information includes: Get the number of bytes corresponding to the file fingerprint; collect the offset; The file status information is generated through the basic information, the fingerprint information, the number of bytes corresponding to the file fingerprint, and the collection offset.

6. The method according to claim 4, wherein Saving the file status information to memory includes: Save the file status information into a map structure with the file path as the key and the file status information as the value; The file status information is saved in a status file whose file name is the inode number.

7. The method according to claim 1, wherein Add event listeners for each target directory and its parent directory corresponding to the file path, including: Add IN_MODIFY event listener and IN_DELETE_SELF event listener to the target directory corresponding to each file path; Add IN_MOVED_TO event listeners and IN_CREATE event listeners for the parent directory corresponding to each file path and its configured directory path; Add an IN_DELETE event listener for the configured target directory.

8. The method according to claim 1, wherein When a log file meets the log rotation condition, log rotation file collection is performed through the event monitoring and the file status information, including: Set the log rotation conditions and rotation mode, which include create rotation mode and copytruncate rotation mode; The log rotation program runs in parallel with the log collection program; Log rotation files are collected through the event monitoring and the file status information.

9. The method according to claim 8, wherein Collecting log rotation files through the event monitoring and the file status information includes: The log file to be saved is identified through the event monitoring and the file status information and saved.

10. A device for collecting Linux system log rotation files, characterized in that: include: Configuration module, used to configure the file path of the log rotation file to be collected in the Linux system; A startup module, configured to obtain a target log file and / or target directory based on the file path when the log collection program is started; A saving module, configured to generate and save file status information of the target log file; A monitoring module is used to add event monitoring for each target directory and its parent directory corresponding to the file path; The collection module is used to collect log rotation files through the event monitoring and the file status information when the log file meets the log rotation condition.

Images