Method for realizing log analysis auxiliary marking in software architecture

Abstract

The invention discloses an auxiliary marking method for realizing log analysis in a software architecture. The auxiliary marking method comprises the following steps: transforming an open source log component, and customizing and developing a unified log output interceptor; the probe agent component runs along with starting of the system and loads configuration parameters under a specified directory file; and the application dynamically modifies the log information, and in the system operation, if more log information is expected to be output, the log file is modified, and the service is restarted to take effect. And by introducing an auxiliary marking function, the richness and the constitutive property of log information are remarkably improved, so that log analysis becomes more efficient and accurate. Through combination of customized log component modification and agent proxy technologies, real-time monitoring and intelligent label classification of key operation events are realized, and the process of problem diagnosis and performance optimization is greatly simplified.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of log analysis, and particularly relates to a method for implementing auxiliary marking of log analysis in a software architecture. BACKGROUND

[0002] In modern software development and system operation processes, log recording is a crucial tool. It not only helps developers and operators track the running state of the system, but also provides key information for troubleshooting when problems occur. However, with the increase in system complexity, the amount of log data also grows exponentially, and traditional log analysis methods face great challenges.

[0003] Most systems use simple timestamps and event levels to organize log entries. This method, although simple, is inefficient when dealing with large amounts of log data, especially when it is necessary to quickly locate specific events or problems. When multiple related events occur within a short period of time, relying solely on timestamps makes it difficult to accurately associate these events, thereby increasing the difficulty of problem diagnosis.

[0004] Prior art solutions:

[0005] Standard format log output strategy

[0006] This strategy adopts a standard log output method. The system stores and outputs formatted log information based on a pre-set log framework. This approach allows logs to be recorded according to different error levels (such as DEBUG, INFO, WARN, ERROR, etc.), enabling developers or operators to more flexibly manage and analyze system running states. When the system fails or during routine maintenance, historical operations need to be audited and analyzed, and these locally saved log files can be directly accessed to obtain the required detailed information. This method not only helps quickly locate the root cause of the problem and improves fault handling efficiency, but also provides valuable data support for subsequent system optimization. In addition, by reasonably configuring the log level, the size and growth rate of the log file can be effectively controlled, avoiding the occupation of too much disk space and affecting server performance. In summary, adopting a standardized log management mechanism is the most commonly used solution for systems.

[0007] Log collection and formatting strategy

[0008] The core of the strategy is to use log collection components to collect logs from different applications on different servers. These log collection components will be deployed on each server, responsible for real-time monitoring and capturing various log information generated by the application. In this way, whether it is a system-level error report or a user operation behavior record, it can be captured completely.

[0009] The collected raw log data is then sent to a tool specifically designed to process log formats. This tool's primary function is to standardize data from various sources, including removing redundant information, converting timestamp formats, and parsing specific fields, to facilitate more efficient and accurate analysis. It also supports custom rule settings, allowing users to tailor data processing to their specific needs for optimal results.

[0010] After formatting, logs are sent to one or more designated storage devices. "Storage" here isn't limited to traditional hard drives or database management systems; it can also include modern solutions like elastic file systems and object storage provided by cloud services. Choosing the right storage solution is crucial to ensuring data security, availability, and scalability.

[0011] All organized and properly stored logs can be accessed through a centralized query platform. This platform usually has powerful search capabilities and a user-friendly interface, allowing administrators to quickly locate problems or generate various statistical reports as needed.

[0012] Disadvantages of existing technology:

[0013] Disadvantages of the standard format log output strategy:

[0014] It is difficult to dynamically categorize the logs of running programs using standard log output formats, especially when various abnormal situations may occur during system operation. It is difficult to dynamically add log tags and categorize these logs.

[0015] Locally stored log files may be subject to disk space limitations. As the system runs longer, log files accumulate, taking up a significant amount of disk space. Failure to clean up or archive old log files promptly can lead to insufficient disk space, impacting server performance or even causing a system crash.

[0016] Disadvantages of log collection and formatting strategies

[0017] This solution has a significant limitation: it can only passively collect pre-defined log information. Regardless of any abnormal conditions or special events that occur during system operation, this solution cannot dynamically add or adjust system tags based on the actual situation. Although collected log data can be categorized and summarized for subsequent analysis and querying, this approach still cannot change the static nature of the source log output.

[0018] Since the log output of the source cannot be dynamically modified, this greatly limits the flexibility and adaptability of this solution. In actual applications, the system may encounter various unexpected situations, and the log recording strategy and content need to be adjusted according to the real-time running state. SUMMARY

[0019] In view of the above problems, the present application is proposed in order to provide a log analysis auxiliary marking method in a software architecture to overcome the above problems or at least partially solve the above problems.

[0020] According to one aspect of the present application, a log analysis auxiliary marking method in a software architecture is provided, and the auxiliary marking method comprises:

[0021] The open source log component is modified, and a unified log output interceptor is developed;

[0022] The probe agent component runs when the system starts and loads the configuration parameters under the specified directory file;

[0023] The log information is dynamically modified, and if more log information is expected to be output during system operation, the log file is modified and the service is restarted to take effect.

[0024] Optionally, the open source log component modification and the development of a unified log output interceptor specifically include that when the system outputs the log, the interceptor intercepts the log and appends the dynamic parameter reserved position of the log output, and the original log is added and output.

[0025] Optionally, the configuration parameters specifically include alarm classification, abnormal level information and key field information.

[0026] Optionally, the open source log component modification specifically includes:

[0027] The open source log component is modified, and a unified log output interceptor is developed;

[0028] After capturing the log, the interceptor performs a series of operations, the most important of which is to append the dynamic parameter reserved position of the log output;

[0029] The interceptor adds the original log, combines the appended dynamic parameter reserved position information with the original log content, and forms a new and more rich log record;

[0030] The new log record is output to the specified log storage medium.

[0031] Optionally, the log storage medium specifically includes a file, a database or a remote log server.

[0032] Optionally, the probe agent component specifically comprises:

[0033] The component has the capability of automatically running at system startup, and loads the files in the specified directory to obtain configuration parameters;

[0034] When the system is running, relevant information is extracted from the running variables of the system based on the previously loaded configuration parameters;

[0035] Data matching the configuration parameters is obtained from the running variables, and the data is appended to the log output;

[0036] The user can understand the running state of the system and the change of relevant parameters by checking the log.

[0037] Optionally, the application dynamically modifies log information specifically comprises:

[0038] Continuously monitor the directory or file system event where the configuration file is located, and when detecting that the configuration file is modified, immediately parse the changes and automatically apply to the current log system without manual intervention or service restart;

[0039] By implementing the dynamic marking function of log classification;

[0040] The agent helps the administrator to distinguish different types of log information.

[0041] Optionally, the configuration file is modified specifically comprises: adding a new log level, changing the log format, or updating the log path.

[0042] Optionally, the agent helps the administrator to distinguish different types of log information specifically comprises: clearly distinguishing error logs, access logs, and transaction logs.

[0043] The application provides a log analysis auxiliary marking method in a software architecture, which comprises: open source log component modification, custom development of a unified log output interceptor; a probe agent component that runs when the system starts and loads configuration parameters under the specified directory file; application dynamically modifies log information, and if more log information is desired during system operation, modify the log file and restart the service to take effect. By introducing the auxiliary marking function, the richness and structure of the log information are significantly improved, making log analysis more efficient and accurate. The combination of customized log component modification and agent technology realizes real-time monitoring and intelligent label classification of key running events, greatly simplifying the problem diagnosis and performance optimization process.

[0044] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, and can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 The flow chart of the log analysis auxiliary marking method implemented in the software architecture provided by the embodiment of the present application is shown in the following figure.

[0047] Figure 2 The flow chart of the open source log component modification provided by the embodiment of the present application is shown in the following figure.

[0048] Figure 3 The flow chart of the probe agent component provided by the embodiment of the present application is shown in the following figure.

[0049] Figure 4 The flow chart of the application dynamic modification of log information provided by the embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] The terms "include" and "have" and any variations thereof in the specification embodiments of the present application and claims and drawings are intended to cover non-exclusive inclusion, for example, including a series of steps or units.

[0052] The technical solutions of the present application will be described in further detail below in conjunction with the drawings and embodiments.

[0053] The software is divided into three parts, the first part is open source log component modification, the second part is probe agent component, and the third part is application dynamic modification of log information.

[0054] Open-source log component modification: The open-source log component is modified, and a unified log output interceptor is developed. When the system outputs logs, the interceptor intercepts the logs and appends the dynamic parameter reserved position of the log output. The original log is added and output.

[0055] Probe agent component: This component can run when the system starts and load configuration parameters in the specified directory file. These parameters generally include alarm classification, exception level information, key field information, etc. When the system runs, the component can obtain related parameters from the running variables in the system based on the loaded configuration parameters and append them to the log output.

[0056] Dynamic modification of log information: In system operation, if more log information is needed, the log file needs to be modified and the service needs to be restarted to take effect. The agent component of this scheme can dynamically detect changes in configuration files and take effect in real time, allowing the system's log classification to be dynamically marked without restarting the system.

[0057] Open-source log component modification:

[0058] The open-source log component is modified, and a unified log output interceptor is developed. The main function of this interceptor is to intercept and process logs when the system outputs logs.

[0059] After capturing the logs, the interceptor performs a series of operations, the most important of which is to append the dynamic parameter reserved position of the log output. These dynamic parameter reserved positions can be configured according to actual needs, so that additional information can be easily inserted or customized processing can be performed during subsequent log analysis or processing.

[0060] After the above operations are completed, the interceptor adds the original log, that is, combines the appended dynamic parameter reserved position information and the original log content to form a new, more comprehensive log record. This new log record will be output to the specified log storage medium, such as a file, a database, or a remote log server.

[0061] Probe agent component:

[0062] The component has the ability to automatically run when the system starts and can load files in a specified directory to obtain configuration parameters. These configuration parameters usually include alarm classification, exception level information, and key field information, as well as other key data.

[0063] When the system is running, the component extracts relevant information from the system's running variables based on the previously loaded configuration parameters. It obtains data matching the configuration parameters from the running variables and appends these data to the log output. Users can understand the running status of the system and the changes of relevant parameters by viewing the log.

[0064] Application of dynamic modification log information:

[0065] The agent component can continuously monitor the directory or file system events where the configuration file is located. Once it detects any modification of the configuration file (such as adding new log levels, changing log formats, or updating log paths, etc.), it parses the changes and automatically applies them to the current log system without manual intervention or service restart. This mechanism greatly improves the efficiency and flexibility of system maintenance, allowing operations personnel to quickly adjust log strategies according to actual needs, while ensuring the high availability and stability of the system.

[0066] By implementing the dynamic marking function of log classification, the agent can also help administrators distinguish different types of log information more meticulously, such as distinguishing error logs, access logs, transaction logs, etc., to facilitate subsequent log analysis and troubleshooting. Throughout the process, unnecessary service restarts are avoided, so there is no interference with ongoing business processing, achieving efficient and intelligent upgrading of log management.

[0067] Benefits:

[0068] Richness and structure: This scheme significantly improves the richness and structure of log information by introducing auxiliary marking functions, making log analysis more efficient and accurate. The combination of customized log component modification and agent proxy technology realizes real-time monitoring and intelligent label classification of key running events, greatly simplifying the problem diagnosis and performance optimization process.

[0069] Flexibility and adaptability: The support for custom marking strategies enhances the flexibility and adaptability of the system, meeting the individual needs of different developers for log analysis.

[0070] The above detailed description further explains the purpose, technical solutions, and benefits of the present application. It should be understood that the above is only a specific implementation of the present application and does not limit the scope of protection. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of protection of the present application.

Claims

1. A log analysis auxiliary marking method implemented in a software architecture, characterized in that: The auxiliary marking method comprises: Transform open source log components and customize and develop a unified log output interceptor; The probe agent component runs when the system starts and loads the configuration parameters in the specified directory file; The application dynamically modifies log information. If you want to output more log information during system operation, modify the log file and restart the service to make it take effect.

2. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The transformation of the open source log component and the customized development of a unified log output interceptor specifically include: When the system outputs logs, the interceptor will intercept the logs, append the dynamic parameter reserved position for log output, and output the original logs after adding them.

3. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The configuration parameters specifically include: alarm classification, abnormality level information and key field information.

4. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The transformation of the open source log component specifically includes: Transform the open source log component and customize and develop a unified log output interceptor; After capturing the log, the interceptor will perform a series of operations, the most important of which is to append the dynamic parameter reserved position of the log output; The interceptor adds to the original log, merging the additional dynamic parameter reserved position information with the original log content to form a new, richer log record; New log records are output to the specified log storage medium.

5. The log analysis auxiliary marking method implemented in a software architecture according to claim 4, characterized in that: The log storage medium specifically includes: a file, a database or a remote log server.

6. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The probe agent component specifically includes: The component has the ability to run automatically when the system starts, and load files in the specified directory to obtain configuration parameters; When the system is running, relevant information is extracted from the system's operating variables based on the previously loaded configuration parameters; Get data that matches the configuration parameters from the running variables and append the data to the log output; Users can view logs to understand the system's operating status and changes in related parameters.

7. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The application dynamic modification log information specifically includes: Continuously monitor events in the directory or file system where the configuration file is located. When a configuration file modification is detected, the system immediately analyzes the change and automatically applies it to the current log system without manual intervention or restarting the service. By implementing dynamic tagging function for log classification; The agent helps administrators distinguish different types of log information.

8. The log analysis auxiliary marking method implemented in a software architecture according to claim 7, characterized in that: The configuration file may be modified by adding a new log level, changing the log format, or updating the log path.

9. The log analysis auxiliary marking method implemented in a software architecture according to claim 1, characterized in that: The agent helps the administrator distinguish different types of log information, specifically by clearly distinguishing error logs, access logs, and transaction logs.

Images