Linux user and program level log automatic rotation method
By dividing the logs into sets of different priorities and setting up personalized rotation cycles, the problem of missing important logs in the existing technology is solved, and more accurate and reliable log management is achieved.
Patent Information
- Application Number
- CN202411544018.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art can easily lead to the lack of important logs when the logs are automatically rotated, and it is impossible to effectively save important logs.
By dividing the logs into sets of different priorities and setting personalized automatic rotation cycles according to priorities, we ensure that high-priority logs are updated more frequently and retained for longer periods of time.
It effectively avoids the loss of important logs, ensures the accuracy and reliability of log management, and avoids waste of storage space due to fixed rotation time.
Smart Images
Figure CN119719063A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method for automatically rotating Linux user and program-level logs. Background Art
[0002] With the continuous development of science and technology, people have higher requirements for the issuance and processing. Automatic rotation of system logs is an efficient log management strategy that aims to optimize the use of storage space by regularly archiving and cleaning log files, while ensuring the availability and integrity of log data. Related technologies generally use a fixed rotation time to automatically rotate logs. The system can automatically delete logs outside the rotation time and manage newly generated logs according to the rotation time. However, when the related technologies automatically rotate logs, important logs may be missing.
[0003] Therefore, how to save important logs during automatic rotation has become a technical problem that those skilled in the art need to solve urgently. Summary of the invention
[0004] Based on the above problems, the present application provides a Linux user and program-level log automatic rotation method to save important logs when the logs are automatically rotated.
[0005] The present application provides a method for automatically rotating Linux user and program-level logs, the method comprising:
[0006] Get multiple logs in the storage space and the log time corresponding to each log;
[0007] Classifying the plurality of logs to obtain a first-category log set, a second-category log set, and a third-category log set, each log set having a corresponding priority;
[0008] Setting a corresponding automatic rotation period for each log set according to the priority;
[0009] Deleting the logs whose log time is not within the automatic rotation period in the first type log set, the second type log set and the third type log set;
[0010] The logs are automatically rotated according to the automatic rotation cycle.
[0011] In a possible implementation, the method for obtaining multiple logs in the storage space and the log time corresponding to each log includes:
[0012] Capture multiple logs from application servers, databases, network devices, and operating system storage spaces;
[0013] Parsing the plurality of logs to obtain parsing results, wherein the parsing results include a timestamp, a log level, a log content, and an operation source;
[0014] A log time of each of the logs is determined based on the timestamp.
[0015] In a possible implementation, the priority is determined in the following manner:
[0016] Determine a basic weight of each log set based on the log level, the log level includes an information level, a warning level and an error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level;
[0017] Determine a weight coefficient of each log set based on an operation source of each log in the log set, wherein the operation source includes a user and a system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user;
[0018] The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
[0019] The present application also provides a Linux user and program level log automatic rotation device, the device comprising:
[0020] An acquisition module is used to acquire multiple logs in the storage space and the log time corresponding to each log;
[0021] A classification module, used for classifying the plurality of logs to obtain a first-category log set, a second-category log set and a third-category log set, each log set having a corresponding priority;
[0022] A setting module, used to set a corresponding automatic rotation period for each log set according to the priority;
[0023] A deleting module, used for deleting logs in the first type of log set, the second type of log set and the third type of log set whose log time is not within the automatic rotation period;
[0024] The rotation module is used to automatically rotate the logs according to the automatic rotation cycle.
[0025] In a possible implementation, the acquisition module is specifically used to:
[0026] Capture multiple logs from application servers, databases, network devices, and operating system storage spaces;
[0027] Parsing the plurality of logs to obtain parsing results, wherein the parsing results include a timestamp, a log level, a log content, and an operation source;
[0028] A log time of each of the logs is determined based on the timestamp.
[0029] In a possible implementation manner, the priority is determined by a priority determination module, and the priority determination module is specifically configured to:
[0030] Determine a basic weight of each log set based on the log level, the log level includes an information level, a warning level and an error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level;
[0031] Determine a weight coefficient of each log set based on an operation source of each log in the log set, wherein the operation source includes a user and a system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user;
[0032] The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
[0033] The present application also provides an electronic device, the electronic device comprising a processor and a memory:
[0034] The memory is used to store a computer program and transmit the computer program to the processor;
[0035] The processor is used to execute the steps of the above-mentioned Linux user and program-level log automatic rotation method according to the instructions in the computer program.
[0036] The present application also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by an electronic device, the steps of the above-mentioned Linux user and program-level log automatic rotation method are implemented.
[0037] Compared with the prior art, this application has the following beneficial effects:
[0038] The method provided by the present application divides logs into first-class, second-class and third-class log sets, and assigns different priorities to each set. This classification method allows the system to formulate more accurate management strategies according to the importance and purpose of the logs. In contrast, the prior art usually adopts a unified fixed rotation time, which cannot perform differentiated processing for different types of logs, which may cause important logs to be deleted by mistake. According to the priority of each log set, the corresponding automatic rotation cycle is set, so that important logs with high priority can be updated more frequently and retained for a longer time, while logs with low priority can be set with a longer rotation cycle to save resources. The fixed rotation time in the prior art cannot be flexibly adjusted, which easily leads to the premature deletion of important logs or the low-importance logs occupying too much storage space. The present application ensures that only logs that are not within the automatic rotation cycle are deleted by performing a double check of the priority and cycle before deleting the log, especially strictly protecting the high-priority logs to prevent the loss of important logs. This mechanism effectively solves the problem of missing important logs caused by fixed rotation in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0040] Figure 1 A flowchart of a Linux user and program-level log automatic rotation method provided in an embodiment of the present application;
[0041] Figure 2 A schematic diagram of the structure of a Linux user and program-level log automatic rotation device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] As described above, the related art generally adopts a method of setting a fixed rotation time for automatic rotation of logs. The system can automatically delete logs outside the rotation time and manage newly generated logs according to the rotation time. However, the related art may cause the loss of important logs when the logs are automatically rotated.
[0043] After research, it was found that in the process of automatic log rotation, the relevant technology completely handed over the management of logs to the system after setting the rotation cycle. For logs, some logs are more important, such as error logs, while some are less important, such as operation logs. The relevant technology did not distinguish between logs, but directly processed all logs in the same way according to the rotation cycle, which led to the loss of some important logs.
[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0045] It is understandable that the method provided by the present application can be applied to a processing device, which is a processing device that can obtain multiple logs in the storage space and the log time corresponding to each log, such as a terminal device or server that can classify multiple logs. The method provided by the present application can be executed independently by a terminal device or a server, and can also be applied to a network scenario in which a terminal device and a server communicate, and is executed by the cooperation of a terminal device and a server. Among them, the terminal device can be a computer, a mobile phone and other devices. The server can be understood as an application server or a Web server. In actual deployment, the server can be a stand-alone server or a cluster server.
[0046] Figure 1 A flowchart of a Linux user and program-level log automatic rotation method provided in an embodiment of the present application, the method comprising the following steps:
[0047] S101: Acquire multiple logs in a storage space and the log time corresponding to each log.
[0048] The processing device may obtain multiple logs in the storage space and the log time corresponding to each log.
[0049] Logging is a key component in the operation of software systems and applications. It provides detailed recording and analysis capabilities of system behaviors. Obtaining multiple logs in the storage space and their corresponding log times is a basic step in log management. In this process, the system will collect logs from multiple sources, including application servers, databases, network devices, operating systems, etc. Each log entry usually contains a timestamp, log level (such as information, warning, error), message content, and possible context information (such as user ID, session ID). The system needs to perform preliminary screening and preprocessing on these logs to ensure the accuracy and effectiveness of subsequent processing. To this end, the system may need to use specific log collection tools or platforms (such as ELK Stack, Splunk, etc.) to import these logs into the log management system in a standardized format. At the same time, the accuracy of log time is crucial. It is not only used to record the time of occurrence of events, but also used for subsequent log classification and rotation cycle judgment. Therefore, it is necessary to ensure the effectiveness of the time synchronization mechanism (such as NTP) to avoid log processing errors caused by time deviation.
[0050] In a possible implementation, the processing device may obtain multiple logs from the storage space of the application server, database, network device, and operating system. The multiple logs are parsed to obtain parsing results, which include timestamps, log levels, log contents, and operation sources. The log time of each log is determined based on the timestamp.
[0051] S102: Categorize multiple logs.
[0052] The processing device may classify the multiple logs to obtain a first-category log set, a second-category log set, and a third-category log set, each of which has a corresponding priority.
[0053] Log classification is a key step in log management, which aims to systematically organize logs according to their content, source and importance. By classifying logs, the efficiency and accuracy of log processing can be improved. In this application, logs are divided into first-class log sets, second-class log sets and third-class log sets. The classification criteria can be based on the source of the log (such as system logs, application logs, security logs), the severity of the log (such as errors, warnings, information) and the purpose of the log (such as auditing, troubleshooting, performance monitoring). Each log set is given different priorities according to its importance and purpose, so as to facilitate subsequent management and processing. Priority division can help the system prioritize and retain more important logs when resources are limited. For example, security-related logs may be classified as the first-class log set with the highest priority to ensure that the required information can be obtained in time when a security incident occurs. In this way, the system can manage massive log data in a more organized way, improve processing efficiency, and provide support for subsequent analysis and decision-making.
[0054] In a possible implementation, the processing device can determine the basic weight of each log set based on the log level, the log level includes information level, warning level and error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level. The weight coefficient of each log set is determined based on the operation source of each log in the log set, the operation source includes user and system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user. The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
[0055] S103: Setting a corresponding automatic rotation period for each log set according to the priority.
[0056] The processing device can set a corresponding automatic rotation period for each log set according to the priority.
[0057] Setting the corresponding automatic rotation cycle according to the priority of the log collection is a key step to ensure the effective operation of the log management system. The setting of the automatic rotation cycle needs to consider multiple factors, including the importance of the log, the availability of storage space, and business needs. For log collections with higher priority, such as security logs and error logs, a shorter rotation cycle may be required to update and back up these key data in a timely manner, so that they can be traced and analyzed quickly when an incident occurs. On the contrary, for log collections with lower priority, such as debug information or general information logs, a longer rotation cycle can be set to save storage space and reduce system burden. When setting the rotation cycle, the log generation rate and data volume also need to be considered to ensure the rational use of storage resources. In some cases, the system can also dynamically adjust the rotation cycle based on real-time monitoring and analysis results to adapt to business needs and environmental changes. This flexible cycle setting not only improves the system's responsiveness, but also effectively optimizes the use of storage resources.
[0058] S104: Delete the logs whose log time is not within the automatic rotation period in the first type log set, the second type log set and the third type log set.
[0059] The processing device may delete the logs in the first type log set, the second type log set and the third type log set whose log time is not within the automatic rotation cycle.
[0060] Deleting logs that are not in the automatic rotation cycle is an important measure to optimize storage space and improve system efficiency. When logs are classified and the corresponding rotation cycle is set, the system will regularly check the timestamps of the logs in each log collection and compare them with the current rotation cycle. For those logs whose time is not in the rotation cycle, the system will automatically delete them to free up storage space. This process is usually completed by automated scripts or log management tools to ensure accuracy and consistency. When deleting logs, the system also needs to pay attention to the backup and archiving strategies of the logs to avoid accidentally deleting critical data. In some cases, administrators can set specific log retention policies to ensure that important historical data can be restored when needed. In addition, the system can further optimize the use of storage resources by compressing logs and archiving them to cold storage. In this way, the system can not only maintain the health of storage space, but also improve the efficiency of data retrieval and analysis.
[0061] S105: Automatically rotate the logs according to the automatic rotation cycle.
[0062] The processing device can automatically rotate the logs according to the automatic rotation cycle.
[0063] By rotating logs regularly, the system can ensure the latest and availability of log data. When performing automatic log rotation, the system automatically archives and replaces log files according to the rotation cycle of each log collection. Archived logs are usually stored in a designated archive directory or storage device so that they can be traced back and analyzed when needed. During the rotation process, the system needs to ensure the integrity and consistency of the log files, which is usually achieved through atomic operations to prevent log data from being damaged or lost during the rotation process. In some cases, the system can also combine operations such as log compression and index updates to improve the efficiency of log storage and retrieval. In addition, automatic rotation also provides a basis for system log monitoring and analysis, ensuring that administrators can obtain the latest system status and behavior information in a timely manner to make accurate decisions. Through automatic log rotation, the system not only improves the efficiency of log management, but also enhances the ability to respond to abnormal events.
[0064] The method provided by the present application divides logs into first-class, second-class and third-class log sets, and assigns different priorities to each set. This classification method allows the system to formulate more accurate management strategies according to the importance and purpose of the logs. In contrast, the prior art usually adopts a unified fixed rotation time, which cannot perform differentiated processing for different types of logs, which may cause important logs to be deleted by mistake. According to the priority of each log set, the corresponding automatic rotation cycle is set, so that important logs with high priority can be updated more frequently and retained for a longer time, while logs with low priority can be set with a longer rotation cycle to save resources. The fixed rotation time in the prior art cannot be flexibly adjusted, which easily leads to the premature deletion of important logs or the low-importance logs occupying too much storage space. The present application ensures that only logs that are not within the automatic rotation cycle are deleted by performing a double check of the priority and cycle before deleting the log, especially strictly protecting the high-priority logs to prevent the loss of important logs. This mechanism effectively solves the problem of missing important logs caused by fixed rotation in the prior art.
[0065] The present application also provides a Figure 2 The schematic diagram of the structure of the Linux user and program level log automatic rotation device shown in the figure, the Linux user and program level log automatic rotation device 200 includes the following modules:
[0066] The acquisition module 201 is used to acquire multiple logs in the storage space and the log time corresponding to each log;
[0067] A classification module 202, used for classifying the plurality of logs to obtain a first type log set, a second type log set and a third type log set, each log set having a corresponding priority;
[0068] A setting module 203, used to set a corresponding automatic rotation period for each log set according to the priority;
[0069] A deleting module 204, configured to delete the logs in the first type of log set, the second type of log set and the third type of log set whose log time is not within the automatic rotation period;
[0070] The rotation module 205 is used to automatically rotate the logs according to the automatic rotation cycle.
[0071] In a possible implementation, the acquisition module is specifically used to:
[0072] Capture multiple logs from application servers, databases, network devices, and operating system storage spaces;
[0073] Parsing the plurality of logs to obtain parsing results, wherein the parsing results include a timestamp, a log level, a log content, and an operation source;
[0074] A log time of each of the logs is determined based on the timestamp.
[0075] In a possible implementation manner, the priority is determined by a priority determination module, and the priority determination module is specifically configured to:
[0076] Determine a basic weight of each log set based on the log level, the log level includes an information level, a warning level and an error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level;
[0077] Determine a weight coefficient of each log set based on an operation source of each log in the log set, wherein the operation source includes a user and a system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user;
[0078] The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
[0079] The device provided by the present application divides logs into first-class, second-class and third-class log sets, and assigns different priorities to each set. This classification method allows the system to formulate more accurate management strategies according to the importance and purpose of the logs. In contrast, the prior art usually adopts a unified fixed rotation time, which cannot perform differentiated processing for different types of logs, which may cause important logs to be deleted by mistake. According to the priority of each log set, the corresponding automatic rotation cycle is set, so that important logs with high priority can be updated more frequently and retained for a longer time, while logs with low priority can be set with a longer rotation cycle to save resources. The fixed rotation time in the prior art cannot be flexibly adjusted, which easily leads to the premature deletion of important logs or the low-importance logs occupying too much storage space. The present application ensures that only logs that are not within the automatic rotation cycle are deleted by performing a double check of the priority and cycle before deleting the log, especially strictly protecting the high-priority logs to prevent the loss of important logs. This mechanism effectively solves the problem of missing important logs caused by fixed rotation in the prior art.
[0080] An embodiment of the present application also provides a Linux user and program-level log automatic rotation device, wherein the device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device performs the steps of the Linux user and program-level log automatic rotation method described in any embodiment of the present application.
[0081] In practical applications, the computer-readable storage medium may be any combination of one or more computer-readable media, which may be a computer-readable signal medium or a computer-readable storage medium.
[0082] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.
[0083] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0084] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0085] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0086] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A Linux user and program level log automatic rotation method, characterized in that: include: Get multiple logs in the storage space and the log time corresponding to each log; Classifying the plurality of logs to obtain a first-category log set, a second-category log set, and a third-category log set, each log set having a corresponding priority; Setting a corresponding automatic rotation period for each log set according to the priority; Deleting the logs whose log time is not within the automatic rotation period in the first type log set, the second type log set and the third type log set; The logs are automatically rotated according to the automatic rotation cycle.
2. The method according to claim 1, characterized in that The method for obtaining multiple logs in the storage space and the log time corresponding to each log includes: Capture multiple logs from application servers, databases, network devices, and operating system storage spaces; Parsing the plurality of logs to obtain parsing results, wherein the parsing results include a timestamp, a log level, a log content, and an operation source; A log time of each of the logs is determined based on the timestamp.
3. The method according to claim 2, characterized in that The priority is determined as follows: Determine a basic weight of each log set based on the log level, the log level includes an information level, a warning level and an error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level; Determine a weight coefficient of each log set based on an operation source of each log in the log set, wherein the operation source includes a user and a system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user; The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
4. A Linux user and program level log automatic rotation device, characterized in that: include: An acquisition module is used to acquire multiple logs in the storage space and the log time corresponding to each log; A classification module, used for classifying the plurality of logs to obtain a first-category log set, a second-category log set and a third-category log set, each log set having a corresponding priority; A setting module, used to set a corresponding automatic rotation period for each log set according to the priority; A deleting module, used for deleting logs in the first type of log set, the second type of log set and the third type of log set whose log time is not within the automatic rotation period; The rotation module is used to automatically rotate the logs according to the automatic rotation cycle.
5. The device according to claim 4, characterized in that The acquisition module is specifically used for: Capture multiple logs from application servers, databases, network devices, and operating system storage spaces; Parsing the plurality of logs to obtain parsing results, wherein the parsing results include a timestamp, a log level, a log content, and an operation source; A log time of each of the logs is determined based on the timestamp.
6. The method according to claim 5, characterized in that The priority is determined by a priority determination module, and the priority determination module is specifically used to: Determine a basic weight of each log set based on the log level, the log level includes an information level, a warning level and an error level, the basic weight of the information level is less than the basic weight of the warning level, and the basic weight of the warning level is less than the basic weight of the book search error level; Determine a weight coefficient of each log set based on an operation source of each log in the log set, wherein the operation source includes a user and a system, and the proportion of the weight coefficient occupied by the system is higher than the proportion of the weight coefficient occupied by the user; The priority corresponding to each log set is determined based on the weight coefficient and the basic weight.
7. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the Linux user and program-level log automatic rotation method as described in any one of claims 1-3.
8. A computer-readable storage medium, characterized in that: Used to store computer programs, wherein when the computer program is executed by a processor, the Linux user and program-level log automatic rotation method as described in any one of claims 1 to 3 is implemented.