Lightweight portable log management analysis method and device and medium

Abstract

The invention relates to the technical field of log analysis, and discloses a lightweight portable log management analysis method and device and a medium, and the method comprises the steps: defining a desensitization performance function and an alarm performance function; packaging the configured log configuration template and the log function class to obtain a log toolkit; configuring the log toolkit into the micro-service; monitoring a change event corresponding to the configuration item in the integrated centralized configuration management service through a log toolkit, and changing the configuration item in the micro-service according to the change event; identifying the log content of the micro-service after the configuration item is changed, and desensitizing the log content through a desensitization function in the log toolkit; triggering an alarm performance function in the log toolkit according to the log level corresponding to the log content, and performing alarm notification on the log content by using the alarm performance function; and outputting the desensitized log content and the log content after the alarm notification to a log output target. According to the invention, the log management efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and particularly to a lightweight portable log management and analysis method, device and medium. Background Art

[0002] In the current era of widespread application of distributed architectures, microservice-based systems are becoming increasingly popular. Each microservice usually needs to independently maintain a log configuration file to define log formats, log storage locations, log retention policies, log styles, log levels, etc. However, due to the large number of services, there are too many problems in independently maintaining the log configurations of each microservice.

[0003] Each microservice may adopt different log configuration formats and policies, making it difficult to achieve global log standardization. For a distributed system, inconsistent log formats will increase the complexity of problem troubleshooting, reduce the availability and analysis efficiency of logs. When it is necessary to adjust the log format or optimize the policy, it is necessary to modify the configuration files of all microservices one by one. Such an operation is not only time-consuming and laborious, but also error-prone. Moreover, some additional log processing functions, such as sensitive keyword desensitization, error log warning, etc., usually need to be implemented separately for each microservice. Such a repetitive implementation method increases the development and maintenance costs and may introduce unnecessary complexity, thus resulting in low efficiency in log management. Summary of the Invention

[0004] The present invention provides a lightweight portable log management and analysis method, device and medium, and its main purpose is to solve the problem of low accuracy in product recommendation.

[0005] To achieve the above object, a lightweight portable log management and analysis method provided by the present invention includes:

[0006] Define a desensitization function and an alarm function in a pre-acquired log framework, and integrate the desensitization function and the alarm function into a log function class;

[0007] Configure the log output target for a predefined log configuration template and the log function class, and package the configured log configuration template and log function class to obtain a log tool package;

[0008] Configure the log tool package into a preset microservice and integrate the log tool package with a preset centralized configuration management service;

[0009] Listen for change events corresponding to configuration items in the integrated centralized configuration management service through the log tool package, and change the configuration items in the microservice according to the change events;

[0010] Identify the log content of the microservice after the configuration item change, and desensitize the log content through the desensitization function in the log toolkit;

[0011] Trigger the alarm function in the log toolkit according to the log level corresponding to the log content, use the alarm function to identify the alarm behavior data corresponding to the log content, and when the alarm behavior data meets the preset behavior alarm condition, give an alarm notification for the log content;

[0012] Output the desensitized log content and the log content after the alarm notification to the log output target.

[0013] Optionally, integrating the desensitization function and the alarm function into a log function class includes:

[0014] Encapsulate the desensitization function and the alarm function;

[0015] Identify the function interfaces of the predefined functions;

[0016] Integrate the encapsulated desensitization function and alarm function into the function through the function interface, and encapsulate the integrated function into a log function class.

[0017] Optionally, configuring the log output target for the predefined log configuration template and the log function class includes:

[0018] Identify the output configuration items in the predefined log configuration template;

[0019] Configure the configuration content corresponding to the output configuration item as the output target of the template content corresponding to the log configuration template;

[0020] Identify the output statements of the functions in the log function class;

[0021] Configure the configuration content corresponding to the output configuration item as the output target of the output statements corresponding to the log function class.

[0022] Optionally, integrating the log toolkit with a preset centralized configuration management service includes:

[0023] Add the dependency information of the centralized configuration management service to the class project corresponding to the log toolkit, and configure the service parameters of the centralized configuration management service in the configuration file of the log toolkit;

[0024] Identify the configuration items in the log toolkit, and create the target configuration items corresponding to the configuration items in the centralized configuration management service;

[0025] Extract the parameter value corresponding to the configuration item through the service interface corresponding to the centralized configuration management service, and load the parameter value into the created target configuration item;

[0026] Integrate the log toolkit with the centralized configuration management service according to the dependency information, the service parameters, and the loaded target configuration item.

[0027] Optionally, the method of listening for the change event corresponding to the configuration item in the integrated centralized configuration management service through the log toolkit includes:

[0028] Initialize the listener in the log toolkit and configure the initialized listener into the centralized configuration management service;

[0029] Filter out the configuration items in the log toolkit from the configuration items of the centralized configuration management service, and use the filtered configuration items as log configuration items;

[0030] Listen for the changed configuration item corresponding to the log configuration item through the listener configured in the centralized configuration management service, and determine the change event according to the changed configuration item.

[0031] Optionally, the method of changing the configuration item in the microservice according to the change event includes:

[0032] Trigger the listener callback function according to the change event;

[0033] Use the listener callback function to parse the configuration change content corresponding to the change event;

[0034] Identify the changed configuration item corresponding to the configuration item in the microservice according to the change event;

[0035] Update the configuration content corresponding to the changed configuration item to the configuration change content.

[0036] Optionally, the method of desensitizing the log content through the desensitization function in the log toolkit includes:

[0037] Extract the character-by-character parsing algorithm and desensitization rules in the desensitization function;

[0038] Convert the log content into a log string, and use the character-by-character parsing algorithm to detect the character position corresponding to the preset starting character in the log string;

[0039] Identify the delimiter in the log string at the character position;

[0040] Determine the sensitive information content in the log content according to the separator, and desensitize the sensitive information content by using the desensitization rule.

[0041] Optionally, the identifying the alarm behavior data corresponding to the log content by using the alarm function includes:

[0042] Extract the alarm rules in the alarm function;

[0043] Match the log content with the alarm list trie in the alarm rules;

[0044] When the match fails, regard the matched log content as the log to be processed, and count the number of logs of the log to be processed and the log accumulation time of all logs to be processed;

[0045] Use the number of logs of the log to be processed and the log accumulation time of all logs to be processed as the alarm behavior data corresponding to the log content.

[0046] To solve the above problems, the present invention further provides an electronic device, and the electronic device includes:

[0047] At least one processor; and,

[0048] A memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the lightweight portable log management and analysis method described above.

[0050] To solve the above problems, the present invention further provides a computer-readable storage medium, and at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the lightweight portable log management and analysis method described above.

[0051] In the embodiments of the present invention, by centrally defining the log configuration template, unified management of the log formats, output locations, retention policies, and log levels of all microservices is achieved, effectively solving the problem of inconsistent log formats in the prior art. According to the dynamic configuration adjustment ability, the cumbersome operation of modifying the configuration file for each microservice separately is avoided. Only one adjustment in the configuration center is required to globally synchronously update the log configuration. The toolkit has built-in extended functions such as sensitive information desensitization and error log warning. Each microservice does not need to develop the corresponding function module repeatedly, reducing code redundancy and development workload, improving the system development efficiency and consistency, simplifying the integration and analysis of log data, facilitating quick problem location in a distributed system, and shortening the problem troubleshooting time. Therefore, the lightweight portable log management and analysis method, device, and medium proposed by the present invention can solve the problem of low efficiency in log management. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 FIG. is a schematic flowchart of a lightweight portable log management and analysis method provided by an embodiment of the present invention;

[0053] Figure 2 FIG. is a schematic diagram of log configuration provided by an embodiment of the present invention;

[0054] Figure 3 FIG. is a schematic flowchart of an alarm notification provided by an embodiment of the present invention;

[0055] Figure 4 FIG. is a schematic structural diagram of an electronic device for implementing the lightweight portable log management and analysis method provided by an embodiment of the present invention.

[0056] The implementation, functional features, and advantages of the objectives of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] An embodiment of the present application provides a lightweight portable log management and analysis method. The execution subject of the lightweight portable log management and analysis method includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the lightweight portable log management and analysis method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0059] Referring to Figure 1 As shown, it is a schematic flowchart of the lightweight portable log management and analysis method provided by an embodiment of the present invention. In this embodiment, the lightweight portable log management and analysis method includes:

[0060] S1. Define a desensitization function and an alarm function in a pre-acquired log framework, and integrate the desensitization function and the alarm function into a log function class.

[0061] In the embodiment of the present invention, the log framework refers to Logback. Logback is a popular Java log framework. The log toolkit is developed based on Logback, and uses the functions and mechanisms provided by Logback to implement operations such as log recording and output, and defines additional log processing capabilities based on the log framework, that is, log desensitization function and log alarm function.

[0062] Specifically, the desensitization function refers to automatically desensitizing sensitive keywords (such as mobile phone numbers, ID card numbers, etc.) in the log content, that is, by customizing a desensitization processing Layout (abbreviation "desensitization tool"), using this tool in all log-level Appenders, the desensitization tool will identify the keyword fields of sensitive information based on predefined rules and perform automated desensitization processing on them; the alarm function is to integrate the alarm function in the Appender at the ERROR level by customizing an alarm Layout (abbreviation "alarm tool"). When the log triggers the ERROR level, it will judge whether it meets the alarm conditions and trigger an alarm. Then, function templates such as sensitive information desensitization and error log alarm are built into the log toolkit to achieve additional log processing capabilities.

[0063] In the embodiments of the present invention, the log function class refers to a class that centrally encapsulates the desensitization function and the alarm function, and is used to perform specific processing operations on logs.

[0064] In the embodiments of the present invention, integrating the desensitization function function and the alarm function function into a log function class includes:

[0065] Encapsulating the desensitization function function and the alarm function function;

[0066] Identifying the function interfaces of predefined function functions;

[0067] Integrating the encapsulated desensitization function function and alarm function function into the function function through the function interface, and encapsulating the integrated function function into a log function class.

[0068] Specifically, the desensitization function function is responsible for processing sensitive information in the log, such as replacing or hiding sensitive data such as ID numbers and bank card numbers through specific algorithms; the alarm function function triggers an alarm operation according to certain rules (such as the log level reaching ERROR). Before integrating the functions, it is necessary to determine the interfaces provided by the function functions. The interface defines the types and quantities of input parameters that the function can accept, as well as the type of the result returned after the function is executed. By identifying the interface, it is possible to clearly know how to call these functions and how to process their return values.

[0069] Specifically, the desensitization function function and the alarm function function are integrated into a class through the function interface, that is, the log function class. For example, if the function interface of the function function is LogFunctionalityImpl, the desensitization function function is desensitize(String logMessage), and the alarm function function is triggerAlert(String logMessage, LogLevel level), then the log function class is LogFunctionalityImpl{desensitize(String logMessage); triggerAlert(String logMessage, LogLevel level);}.

[0070] Furthermore, when it is necessary to adjust the log format or policy, it is necessary to modify the log configuration files of each microservice one by one, which is cumbersome and error-prone. Therefore, it is necessary to unify the log configuration to manage the log output uniformly, and it is necessary to build a log configuration template in the toolkit.

[0071] S2. Configure the log output target for the pre-defined log configuration template and the log function class, and package the configured log configuration template and log function class to obtain a log toolkit.

[0072] In the embodiment of the present invention, the pre-defined log configuration template includes, but is not limited to, the length of a single log, the log output location, the log format, the log retention policy, and the default log level. The log output target refers to the output location corresponding to the log content of the microservice, which is convenient for unified management of log output.

[0073] In the embodiment of the present invention, the configuration of the log output target for the pre-defined log configuration template and the log function class includes:

[0074] Identify the output configuration items in the pre-defined log configuration template;

[0075] Configure the configuration content corresponding to the output configuration item as the output target of the template content corresponding to the log configuration template;

[0076] Identify the output statements of the functional functions in the log function class;

[0077] Configure the configuration content corresponding to the output configuration item as the output target of the output statements corresponding to the log function class.

[0078] Specifically, determine the target type of log output in the log configuration template (whether it is output to the console, file, or other such as database, etc.). According to the configuration items, determine where the log content specified by the log configuration template will be output and in what format. For example, if the output configuration item specifies that the log is to be output to the logs / app.log file and the format is %d{yyyy-MM-dd HH:mm:ss}, [%thread]%-5level%logger{36}-%msg%n, then the log content corresponding to the log configuration template will be output to the specified file in this format.

[0079] Specifically, the output statement refers to the code part in the functional function that generates the output. After identifying the output statements of the functional functions in the log function class, output the content generated by the output statements (such as the desensitized log message) to the output address specified by the configuration item to achieve unified management of all log outputs.

[0080] Furthermore, the toolkit is based on logback. It connects to the logback operation log through the java-api method and finally generates a log toolkit. Then, the configured log configuration template and log function classes are packaged through a packaging tool to obtain the log toolkit. The log toolkit provides centralized definition and distribution of log configurations (standardized log formats, standardized log locations, standardized log file retention policies), dynamic adjustment of log configurations, dynamic adjustment of log levels, and additional log processing capabilities (log alerts, log desensitization). Among them, Apollo is used to dynamically maintain all configurations in the above process, enabling each project to support system-customized configurations.

[0081] Furthermore, as Figure 2 shown, it is a schematic diagram of log configuration. First, initialize the toolkit and the log system, and based on the Spring framework, it means running in the Spring environment. After initialization, start the standard log and remove all existing Appenders (log output destinations, such as the console, files, etc.) to prepare for re-setting according to the configuration. Obtain a series of configuration information, such as application name, maximum length of bank information, and other parameters. These configurations will determine the specific behavior of the log system. Judge whether to enable the desensitization function according to the configuration. If enabled, the Appender loads the desensitized Layout (layout, used to format the log output) to process sensitive information in the log; if not enabled, judge whether to enable asynchronous logging. If enabled, assemble the asynchronous Appender to improve the efficiency of log recording; if not enabled, assemble the logger. The logger is the core component of log recording. After assembling the logger, set the log level, stipulate the log recording standards for different levels of importance, and judge whether the current running environment is the local environment. If so, add stdout (standard output, that is, console output) to facilitate viewing logs during local development; if not, directly end the configuration process.

[0082] S3. Configure the log toolkit into a preset microservice and integrate the log toolkit with a preset centralized configuration management service.

[0083] In the embodiment of the present invention, before the log toolkit is used, each microservice independently maintains the logback.xml file, resulting in difficulties in unifying log formats, output locations, retention policies, etc., making it difficult to standardize log configurations in a distributed system and difficult to uniformly manage the log outputs of all microservices. However, by configuring the developed log toolkit into the microservice, unified management of the log outputs of the microservices is achieved. That is, each microservice only needs to introduce the toolkit, and the toolkit can automatically load the global configuration during initialization and apply it to the current service without independently maintaining the logback.xml file.

[0084] Furthermore, the logging toolkit supports dynamic adjustment of all logging configurations by integrating with a centralized configuration management service (Apollo), including but not limited to the length of a single log, the log output location, the log format, the log retention policy, the default log level, etc. When it is necessary to modify the log format or retention policy, the administrator only needs to update the configuration center, and the toolkit can listen in real time and automatically synchronize the configuration changes without manual intervention. Therefore, to achieve dynamic adjustment of logging configurations, the toolkit is integrated with the centralized configuration management service (Apollo).

[0085] In the embodiments of the present invention, the integration of the logging toolkit with a preset centralized configuration management service includes:

[0086] Adding dependency information of the centralized configuration management service to the class project corresponding to the logging toolkit, and configuring service parameters of the centralized configuration management service in the configuration file of the logging toolkit;

[0087] Identifying configuration items in the logging toolkit, and creating target configuration items corresponding to the configuration items in the centralized configuration management service;

[0088] Extracting parameter values corresponding to the configuration items through a service interface corresponding to the centralized configuration management service, and loading the parameter values into the created target configuration items;

[0089] Integrating the logging toolkit with the centralized configuration management service according to the dependency information, the service parameters, and the loaded target configuration items.

[0090] Specifically, add the dependency of the Apollo client to the toolkit project to ensure that the project can use the APIs (Application Programming Interfaces) provided by Apollo to obtain configuration information, and in the configuration file of the toolkit, configure the necessary parameters of the Apollo client, such as the address of the Apollo configuration center, the App ID, the cluster name, etc.

[0091] Specifically, identify the items in the recognition toolkit that need to be dynamically configured, such as log levels, log output paths, desensitization rules, alarm thresholds, etc., and create corresponding configuration items in the corresponding application namespace of the Apollo management interface, and set default values for each configuration item, so that when the client cannot connect to the Apollo configuration center, the toolkit can still run normally using the default configuration. Furthermore, when the toolkit is started, initialize the Apollo client to establish a connection with the Apollo configuration center, obtain the values of the configuration items from the configuration center through the API of the Apollo client, and apply the obtained values of the configuration items to the corresponding functional modules of the toolkit. For example, according to the obtained log level configuration, adjust the log output level of the toolkit, so as to realize the integration of the log toolkit and the centralized configuration management service based on the dependency information added to the toolkit, the service parameters of the centralized configuration management service, and the configuration items.

[0092] Furthermore, when it is necessary to modify the log format or retention policy, the administrator only needs to update the configuration center, and the toolkit can listen in real time and automatically synchronize the configuration changes without manual intervention, and the microservices can perform personalized configurations according to the configuration items provided by the toolkit.

[0093] S4. Listen for the change events corresponding to the configuration items in the integrated centralized configuration management service through the log toolkit, and change the configuration items in the microservices according to the change events.

[0094] In the embodiments of the present invention, the change event refers to the situation where the configuration items related to the log toolkit (i.e., log configuration items) in the centralized configuration management service change, which can be that the value of the configuration item is modified, or operations such as addition or deletion of the configuration item.

[0095] In the embodiments of the present invention, the listening for the change events corresponding to the configuration items in the integrated centralized configuration management service through the log toolkit includes:

[0096] Initialize the listener in the log toolkit and configure the initialized listener to the centralized configuration management service;

[0097] Filter out the configuration items in the log toolkit from the configuration items of the centralized configuration management service, and use the filtered configuration items as log configuration items;

[0098] Listen for the changed configuration items corresponding to the log configuration items through the listener configured to the centralized configuration management service, and determine the change event according to the changed configuration items.

[0099] Specifically, register a configuration change listener in the toolkit and initialize the listener. When the application starts, register an Apollo configuration change listener. When the configuration items in the Apollo configuration center change, the listener can capture the change event in a timely manner. After the configuration change event is detected, the toolkit needs to update the corresponding function modules according to the changed configuration items. For example, if the log level configuration changes, the toolkit should immediately adjust the log output level.

[0100] Specifically, when the application starts, register an Apollo configuration change listener and subscribe to the configuration items related to logging, such as: log level (e.g., log.level); log retention policy (e.g., log.retentionPolicy); sensitive field rules (e.g., log.sensitive.keys); alarm policy (e.g., log.alarm.batch.size, log.alarm.batch.time); and listen for configuration change events through the listener. When the relevant configuration items in Apollo change, trigger the listener callback function to parse the changed configuration content. After the configuration of the toolkit changes, the relevant configurations corresponding to all microservices will take effect. When the personalized configuration of a microservice changes, only the current microservice will take effect after the change. Thus, determine the change event based on the monitored configuration items, where the change event includes but is not limited to dynamic updates of log level / log retention policy, updates of sensitive field rules, and adjustments of alarm policy.

[0101] Furthermore, according to the change events monitored by the centralized configuration management service, only the configuration center needs to be updated. The toolkit can monitor and synchronize configuration changes in real time automatically without manual intervention.

[0102] In the embodiment of the present invention, changing the configuration items in the microservice according to the change event includes:

[0103] Trigger the listener callback function according to the change event;

[0104] Use the listener callback function to parse the configuration change content corresponding to the change event;

[0105] Identify the changed configuration items corresponding to the configuration items in the microservice according to the change event;

[0106] Update the configuration content corresponding to the changed configuration items to the configuration change content.

[0107] Specifically, when relevant configuration items in Apollo change, a listener callback function is triggered to parse the changed configuration content. The callback function is called when the listener detects a specific event (i.e., configuration item change). After the listener callback function is triggered, it parses the specific information contained in the change event and extracts the configuration change content from it. For example, if the original configuration item was "log level is INFO" and now it becomes "log level is DEBUG", the callback function needs to parse this change from INFO to DEBUG to obtain the corresponding configuration change content of the configuration item.

[0108] Specifically, according to the corresponding changed configuration item in the microservice, the new configuration content parsed from the change event is applied to this configuration item of the microservice. Taking the log level as an example, the original value of the log.level configuration item in the microservice is replaced with the newly parsed value. For example, if it is updated from INFO to DEBUG, the microservice runs according to the latest configuration content, completing the change operation of the configuration item based on the change event.

[0109] Exemplarily, for the dynamic update of the log level / log retention policy, in the listener, the dynamic adjustment interface of the logging framework can be called according to the changed log.level value. For the update of the sensitive field rules, according to the changed log.sensitive.keys value, the rules in the desensitization tool are updated, and the new rules will be immediately applied to the subsequent log processing process. For the adjustment of the alarm policy, according to the changed alarm configuration (such as log.alarm.batch.size), the batch policy in the alarm processor is dynamically adjusted, and the changed configuration will take effect directly without restarting the application. The new configuration immediately affects the subsequent log processing logic. When there is a problem with the new configuration, it is supported to quickly roll back to the old version through the Apollo configuration center, and the listener will reload the rolled-back configuration content to ensure the normal operation of the system.

[0110] Furthermore, microservices often need to implement functions such as sensitive information desensitization and error log alarming, but usually each service needs to be developed independently, resulting in duplicate work. Therefore, an additional function needs to be provided centrally through a unified log management toolkit to solve the problem of redundant function implementation.

[0111] S5. Identify the log content of the microservice after the configuration item change, and desensitize the log content through the desensitization function in the log toolkit.

[0112] In the embodiment of the present invention, the log content is a series of information recorded during the operation of the microservice, used to describe the operation state, operation process, and events that occur in the microservice. Among them, the log content of the microservice after the configuration item change is intercepted through an interceptor.

[0113] Furthermore, the toolkit is built with a keyword desensitization function module that can automatically identify and process sensitive information that may be contained in the logs, such as ID card numbers, names, mobile phone numbers, passwords, etc. During the process of log generation and output, desensitization processing is performed on this sensitive information, and only necessary information fields are retained, thus effectively preventing the risk of privacy information leakage caused by log leakage.

[0114] In the embodiments of the present invention, desensitizing the log content through the desensitization function in the log toolkit includes:

[0115] Extracting the character-by-character parsing algorithm and desensitization rules in the desensitization function;

[0116] Converting the log content into a log string, and using the character-by-character parsing algorithm to detect the character position corresponding to the preset start character in the log string;

[0117] Identifying the delimiter in the log string at the character position;

[0118] Determining the sensitive information content in the log content according to the delimiter, and desensitizing the sensitive information content using the desensitization rules.

[0119] Specifically, by analyzing the log format, for JSON and Java toString formats, the parsing rules for sensitive fields are as follows: Key start character: starting with {, [, (, ignoring double quotes; Value end character: ending with}, ], ), ignoring double quotes; Key / Value delimiter: using = or : as the delimiter. Based on the above rules, a character-by-character parsing algorithm based on a finite state machine (FSM) is proposed, that is, traversing the log string, detecting the start position of the Key, parsing the Key / Value delimiter backward to determine the Key, parsing backward to the end character of the Value to determine the Value, and performing desensitization operations on the Value in the identified sensitive Key / Value pairs. The time complexity of this algorithm is O(n), which can efficiently process large-scale log data, and the definition of sensitive fields is centrally managed by a configuration center (such as Apollo), which is convenient for dynamic updates.

[0120] Specifically, the log content is converted into a character form, and the character positions corresponding to the start characters key and value in the log string are identified through a character-by-character parsing algorithm, and the delimiter is identified at the character position, so as to determine the sensitive information content in the log string, and the sensitive information is desensitized using pre-customized desensitization rules, such as desensitization rules including but not limited to replacement rules and hash rules.

[0121] Further, for logs with the ERROR log level, trigger the alarm mechanism to implement additional log processing capabilities and reduce the repetitive development work of microservices.

[0122] S6. Trigger the alarm function in the log toolkit according to the log level corresponding to the log content, use the alarm function to identify the alarm behavior data corresponding to the log content, and when the alarm behavior data meets the preset behavior alarm condition, send an alarm notification for the log content.

[0123] In the embodiment of the present invention, by customizing the alarm Layout (abbreviated as "alarm tool"), integrating the alarm function in the Appender at the ERROR level, when the log triggers the ERROR level, trigger the alarm function. The system will judge whether it meets the alarm condition and trigger the alarm, and send the alarm information to the alarm processor. The alarm processor batches the logs according to the message accumulation strategy, optimizes the use of network I / O, and improves the system throughput.

[0124] In the embodiment of the present invention, the identifying the alarm behavior data corresponding to the log content by using the alarm function includes:

[0125] Extracting the alarm rules in the alarm function;

[0126] Matching the log content with the alarm list trie in the alarm rules;

[0127] When the matching fails, take the matched log content as the log to be processed, and count the number of logs to be processed and the log accumulation time of all logs to be processed;

[0128] Take the number of logs to be processed and the log accumulation time of all logs to be processed as the alarm behavior data corresponding to the log content.

[0129] When the number of logs reaches the preset number threshold or the log accumulation time exceeds the preset time threshold, trigger an alarm notification for the logs to be processed;

[0130] Asynchronously send the logs to be processed for which the alarm notification is triggered to the alarm processor.

[0131] Specifically, the alarm whitelist keywords are stored in the configuration center, and an alarm list trie tree is generated based on the alarm whitelist keywords. The trie tree (Trie) algorithm is used for fast matching. If the log content contains whitelist keywords, the alarm processing is skipped. When the matching fails, the number of logs of the to-be-processed log and the log accumulation time of all to-be-processed logs are used as the alarm behavior data corresponding to the log content. That is, when the number of logs reaches the preset number threshold or the log accumulation time exceeds the preset time threshold, an alarm notification is triggered for the to-be-processed log, and the to-be-processed log triggering the alarm notification is asynchronously sent to the alarm processor.

[0132] Specifically, to improve the alarm push efficiency and avoid frequent triggering, the alarm messages adopt a batch sending strategy, and the specific rules are as follows: When the number of to-be-processed logs reaches the set threshold log.alarm.batch.size, the message push is triggered; when the accumulated log time exceeds the set threshold log.alarm.batch.time, the message push is triggered, and both conditions take effect simultaneously. The parameters support dynamic configuration. The log component asynchronously sends the logs triggering the alarm to the alarm processor, and the alarm processor processes the logs in batches according to the message accumulation strategy. That is, when the matching fails, the matched key content is used as the to-be-processed log, and the number of logs of all to-be-processed logs or the log time of the to-be-processed logs is counted, and an alarm notification is triggered according to the number of logs and the log time. The specific information of the alarm notification includes but is not limited to the alarm time, alarm type, and reason for triggering the alarm.

[0133] Exemplarily, if log.alarm.batch.size = 10, that is, when the number of to-be-processed logs reaches 10, the message push is triggered. If log.alarm.batch.time = 60, that is, when the accumulated log time exceeds 60 seconds, the message push is triggered. Starting from 2024-12-31 10:00:00, the system has successively generated some logs that need to be alarmed, but the number has not reached 10. At 2024-12-31 10:01:01, the accumulated log time exceeds 60 seconds. At this time, the system checks that the condition of log.alarm.batch.time is met, so all the logs that need to be alarmed accumulated during this period are processed in batches and the alarm message is sent. Another example is that starting from 2024-12-31 10:05:00, new logs that need to be alarmed are generated. At 2024-12-31 10:05:20, the number of to-be-processed logs that need to be alarmed reaches 10. Although the accumulated log time does not exceed 60 seconds at this time, but the condition of log.alarm.batch.size is met, and the system will also process these 10 logs in batches and send the alarm message.

[0134] Furthermore, asFigure 3 As shown, it is a schematic diagram of the alarm notification process. When a situation that requires an alarm occurs, an error log will be printed to record the error-related information. After printing the error log, the relevant information will be sent to the alarm processor asynchronously (async). An async thread subscribes to the alarm-related information, puts the received data into a queue, and determines whether the size of the data in the queue is greater than a pre-set threshold. If the queue size is not greater than the threshold, new alarm information will continue to enter the queue and wait for subsequent processing. If the queue size is greater than the threshold, the operation of "discarding the head of the queue and new data entering the queue" will be executed, that is, the earliest entered alarm information in the queue will be discarded to make room for new alarm information, ensuring that the queue length is maintained within a reasonable range. At the same time, the system will regularly push the alarm information in the queue and clear the queue after pushing, ensuring that the alarm information is processed in a timely manner and not processed repeatedly.

[0135] Furthermore, the toolkit has optimized the design of log output. While providing unified management capabilities, it ensures that the performance overhead of log processing is minimized, thus guaranteeing the stability of the overall performance of the distributed system. Therefore, the log content is uniformly output and managed.

[0136] S7. Output the desensitized log content and the log content after alarm notification to the log output target.

[0137] In the embodiment of the present invention, the log output target refers to a pre-set place for log storage or display. It can be a certain log file in the file system for long-term storage of logs for subsequent reference and analysis. It can also be the console, which is convenient for developers or operators to view the logs in real time during the operation of the system. It may also be a database for structured storage and complex query of logs. Or it may be some specialized log management platforms for centralized management and analysis of logs. Outputting the desensitized log content and the log content after alarm notification to the log output target can achieve unified management of the log output of all microservices. And through centralized management, it solves the problem of low efficiency in log configuration change, and supports dynamic adjustment and rapid release.

[0138] In the embodiments of the present invention, by centrally defining a log configuration template, unified management of the log formats, output locations, retention policies, and log levels of all microservices is achieved, effectively solving the problem of inconsistent log formats in the prior art; and according to the dynamic configuration adjustment capability, the cumbersome operation of separately modifying the configuration files for each microservice is avoided. Only one adjustment in the configuration center is required to globally synchronously update the log configuration; the toolkit has built-in extended functions such as sensitive information desensitization and error log warning. Each microservice does not need to develop the corresponding functional modules repeatedly, reducing code redundancy and development workload, improving the system development efficiency and consistency, simplifying the integration and analysis of log data, facilitating quick problem location in a distributed system, and shortening the time for problem troubleshooting. Therefore, the lightweight portable log management and analysis method, device, and medium proposed by the present invention can solve the problem of low efficiency in log management.

[0139] As Figure 4 shown, it is a schematic structural diagram of an electronic device for implementing the lightweight portable log management and analysis method provided by an embodiment of the present invention.

[0140] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a lightweight portable log management and analysis program.

[0141] Among them, in some embodiments, the processor 10 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as executing a lightweight portable log management and analysis program, etc.), and calling data stored in the memory 11, to perform various functions of the electronic device and process data.

[0142] The memory 11 at least includes one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disks, multimedia cards, card-type memories (such as SD or DX memories, etc.), magnetic memories, magnetic disks, optical disks, etc. The memory 11 can be an internal storage unit of the electronic device in some embodiments, such as the mobile hard disk of the electronic device. The memory 11 can also be an external storage device of the electronic device in some other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software installed in the electronic device and various types of data, such as the code of a lightweight portable log management and analysis program, etc., but also to temporarily store data that has been output or will be output.

[0143] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is set to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0144] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between this electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, and is used to display the information processed in the electronic device and to display a visual user interface.

[0145] Only an electronic device with components is shown in the figure. Those skilled in the art can understand that the structure shown in the figure does not constitute a limitation on the electronic device, and it may include fewer or more components than shown in the figure, or combine some components, or have different component arrangements.

[0146] For example, although not shown, the electronic device may further include a power source (such as a battery) for supplying power to each component. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0147] It should be understood that the above embodiments are only for illustration purposes and are not limited by this structure in the scope of the patent application.

[0148] The lightweight portable log management and analysis program stored in the memory 11 of the electronic device is a combination of multiple instructions. When running in the processor 10, it can implement:

[0149] Define a desensitization function and an alarm function in a pre-acquired log framework, and integrate the desensitization function and the alarm function into a log function class;

[0150] Configure the log output target for a pre-defined log configuration template and the log function class, and package the configured log configuration template and log function class to obtain a log tool kit;

[0151] Configure the log tool kit into a preset microservice and integrate the log tool kit with a preset centralized configuration management service;

[0152] Listen for change events corresponding to configuration items in the integrated centralized configuration management service through the log tool kit, and change the configuration items in the microservice according to the change events;

[0153] Identify the log content of the microservice after the configuration item change, and desensitize the log content through the desensitization function in the log tool kit;

[0154] Trigger the alarm function in the log tool kit according to the log level corresponding to the log content, use the alarm function to identify the alarm behavior data corresponding to the log content, and when the alarm behavior data meets the preset behavior alarm condition, give an alarm notification for the log content;

[0155] Output the desensitized log content and the log content after alarm notification to the log output target.

[0156] Specifically, for the specific implementation method of the above instructions by the processor 10, reference may be made to the description of the relevant steps in the corresponding embodiments of the attached drawings, which will not be elaborated herein.

[0157] Furthermore, if the modules / units integrated in the electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0158] The present invention also provides a computer-readable storage medium, where the readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, it can implement:

[0159] Define a desensitization function and an alarm function in a pre-acquired log framework, and integrate the desensitization function and the alarm function into a log function class;

[0160] Configure the log output target for a pre-defined log configuration template and the log function class, and package the configured log configuration template and log function class to obtain a log tool kit;

[0161] Configure the log tool kit into a preset microservice, and integrate the log tool kit with a preset centralized configuration management service;

[0162] Listen for change events corresponding to configuration items in the integrated centralized configuration management service through the log tool kit, and change the configuration items in the microservice according to the change events;

[0163] Identify the log content of the microservice after the configuration item change, and desensitize the log content through the desensitization function in the log tool kit;

[0164] Trigger the alarm function in the log tool kit according to the log level corresponding to the log content, use the alarm function to identify the alarm behavior data corresponding to the log content, and when the alarm behavior data meets the preset behavior alarm condition, give an alarm notification for the log content;

[0165] Output the desensitized log content and the log content after alarm notification to the log output target.

[0166] In several embodiments provided by the present invention, it should be understood that the disclosed devices, media, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0167] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0169] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0170] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is not limited only by the above description. Therefore, it is intended to include all changes within the meaning and scope of equivalent elements falling within the protection scope in the present invention.

[0171] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain the best results of theory, method, technology, and application system.

[0172] In addition, obviously, the word "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the system can also be implemented by one unit or device through software or hardware. Words such as first and second are used to represent names and do not represent any specific order.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A lightweight portable log management and analysis method, characterized in that, The method includes: Defining a desensitization function and an alarm function in a pre-acquired logging framework, and integrating the desensitization function and the alarm function into a logging function class; Configuring the logging output target for a pre-defined logging configuration template and the logging function class, and packaging the configured logging configuration template and logging function class to obtain a logging toolkit; Configuring the logging toolkit into a preset microservice, and integrating the logging toolkit with a preset centralized configuration management service; Listening for a change event corresponding to a configuration item in the integrated centralized configuration management service through the logging toolkit, and changing the configuration item in the microservice according to the change event; Identifying the log content of the microservice after the configuration item change, and desensitizing the log content through the desensitization function in the logging toolkit; Triggering the alarm function in the logging toolkit according to the log level corresponding to the log content, and using the alarm function to identify the alarm behavior data corresponding to the log content. When the alarm behavior data meets the preset behavior alarm condition, an alarm notification is sent for the log content; Outputting the desensitized log content and the log content after the alarm notification to the logging output target; 2. The lightweight portable log management and analysis method according to claim 1, wherein, The integrating the desensitization function and the alarm function into a logging function class includes: Encapsulating the desensitization function and the alarm function; Identifying the function interfaces of the pre-defined functions; Integrating the encapsulated desensitization function and alarm function into the function through the function interfaces, and encapsulating the integrated function into a logging function class; 3. The lightweight portable log management and analysis method according to claim 1, wherein, The configuring the logging output target for a pre-defined logging configuration template and the logging function class includes: Identifying the output configuration items in the pre-defined logging configuration template; Configuring the configuration content corresponding to the output configuration item as the output target of the template content corresponding to the logging configuration template; Identifying the output statements of the functions in the logging function class; Configuring the configuration content corresponding to the output configuration item as the output target of the output statements corresponding to the logging function class; 4. The lightweight portable log management and analysis method according to claim 1, wherein, The integrating the logging toolkit with a preset centralized configuration management service includes: Adding dependency information of the centralized configuration management service to the class project corresponding to the logging toolkit, and configuring the service parameters of the centralized configuration management service in the configuration file of the logging toolkit; Identifying the configuration items in the logging toolkit, and creating target configuration items corresponding to the configuration items in the centralized configuration management service; Extracting the parameter values corresponding to the configuration items through the service interface corresponding to the centralized configuration management service, and loading the parameter values into the created target configuration items; Integrating the logging toolkit with the centralized configuration management service according to the dependency information, the service parameters and the loaded target configuration items; 5. The lightweight portable log management and analysis method according to claim 1, wherein The listening for a change event corresponding to a configuration item in the integrated centralized configuration management service through the logging toolkit includes: Initialize the listener in the log toolkit and configure the initialized listener into the centralized configuration management service; Filter out the configuration items in the log toolkit from the configuration items of the centralized configuration management service, and use the filtered configuration items as log configuration items; Monitor the changed configuration items corresponding to the log configuration items through the listener configured in the centralized configuration management service, and determine change events according to the changed configuration items.

6. The lightweight portable log management and analysis method according to claim 1, wherein The changing the configuration items in the microservice according to the change event includes: Trigger the listener callback function according to the change event; Use the listener callback function to parse the configuration change content corresponding to the change event; Identify the changed configuration items corresponding to the configuration items in the microservice according to the change event; Update the configuration content corresponding to the changed configuration items to the configuration change content.

7. The lightweight portable log management and analysis method according to claim 1, wherein The desensitizing the log content through the desensitization function in the log toolkit includes: Extract the character-by-character parsing algorithm and desensitization rules in the desensitization function; Convert the log content into a log string, and use the character-by-character parsing algorithm to detect the character position corresponding to the preset starting character in the log string; Identify the delimiter in the log string among the character positions; Determine the sensitive information content in the log content according to the delimiter, and desensitize the sensitive information content using the desensitization rules.

8. The lightweight portable log management and analysis method according to claim 1, characterized in that The identifying the alarm behavior data corresponding to the log content using the alarm function includes: Extract the alarm rules in the alarm function; Match the log content with the alarm list trie in the alarm rules; When the match fails, use the matched log content as the log to be processed, and count the number of logs to be processed and the log accumulation time of all logs to be processed; Use the number of logs to be processed and the log accumulation time of all logs to be processed as the alarm behavior data corresponding to the log content.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the lightweight portable log management and analysis method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, implements the lightweight portable log management and analysis method according to any one of claims 1 to 8.

Images