Method and device for multi-dimensional dynamic filtering and clearing control of system log

Abstract

The application discloses a kind of system log multidimensional dynamic filtering and control method and device, belong to information technology field, the method includes the following steps: receiving log event and obtaining its associated multiple business dimensions business element information and thread context identifier;Log event associated thread context identifier is input into preset single key dynamic threshold filter, and its dynamic rule set is matched and first filtering decision is generated;Business element information is input into preset multikey dynamic threshold filter, and its dynamic rule base is matched and second filtering decision is generated;Based on the cooperative processing result of first filtering decision and second filtering decision, determine final filtering decision;Based on the preset multidimensional log dynamic clearing strategy, according to the attribute characteristics and environment configuration of log event, dynamically determine its output file path.The application realizes the accurate control and efficient retrieval of log output, improves system operation efficiency and problem positioning speed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, and relates to a method and device for multi-dimensional dynamic filtering and clearing control of system logs, which can be used for log output control of systems such as finance, telecommunications and e-commerce. BACKGROUND

[0002] In the core business processing of commercial systems, especially in the financial and banking industries, the system generates a large amount of log data of various types, including transaction tracking logs, transaction message logs, communication logs, database access logs, etc. These logs are crucial for problem troubleshooting, system monitoring, security auditing and compliance checking. With the increasing complexity of business systems and the expansion of system size, the scale and complexity of log data also grow exponentially, and how to efficiently and finely manage these log data has become an important technical challenge in the industry.

[0003] Currently, the main technical problems of commercial system log management are as follows: 1) Single dimension of log filtering control: Traditional log level control is usually global or module level, which cannot achieve fine and multi-dimensional control based on specific business elements (such as specific transaction type, specific institution, specific teller), resulting in the need to turn on extensive detailed logs when troubleshooting specific business problems, generating a large amount of irrelevant logs, which seriously affects troubleshooting efficiency and system performance. 2) Log filtering rules are rigid: Existing log filtering rules are usually loaded at application startup, and modifying the rules requires restarting the application, which cannot meet the needs of quickly and dynamically adjusting log output strategies in production or test environments. 3) Fixed log clearing method: Log files are usually stored in a fixed mode (such as by day, by application), which cannot dynamically adjust the organization of log files according to actual troubleshooting needs (such as tracking complete transaction links by thread, or viewing all operations by teller), resulting in the need to perform tedious screening in a large number of mixed log files when locating problems. 4) Lack of effective collaborative filtering mechanism: Existing solutions usually use a single filtering logic, which fails to organically combine filtering based on specific business content with filtering based on log processing context, making it difficult to achieve precise and efficient log output control in complex business scenarios.

[0004] To address the above technical problems, the present application provides an innovative technical solution that can achieve multi-dimensional, dynamic and fine filtering and clearing of commercial system logs, in order to improve the efficiency and accuracy of log management, reduce the time cost of problem troubleshooting, and improve system operation and maintenance level. SUMMARY

[0005] To solve the above problems, the present application provides a method and device for multi-dimensional dynamic filtering and clearing control of system logs, which can achieve precise control and efficient retrieval of log output, and improve system operation and maintenance efficiency and problem positioning speed.

[0006] The technical scheme adopted by the present application to solve the technical problems is:

[0007] In a first aspect, the embodiment of the present application provides a method for multi-dimensional dynamic filtering and clearing control of system logs, comprising the following steps:

[0008] Step S1, receiving a log event, obtaining business element information of a plurality of business dimensions associated with the log event and a thread context identifier, the business element information including transaction type, transaction name, institution identifier and teller identifier;

[0009] Step S2, inputting the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, matching with a preset single-key dynamic rule set, and generating a first filtering decision according to the matching result;

[0010] Step S3, inputting the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, matching with a preset multi-key dynamic rule set, and generating a second filtering decision according to the matching result;

[0011] Step S4, determining the final filtering decision of the log event based on the cooperative processing result of the first filtering decision and the second filtering decision;

[0012] Step S5, based on the preset multi-dimensional log dynamic clearing strategy, dynamically determining the output file path of the log event according to the attribute characteristics and environment configuration of the log event.

[0013] As a possible implementation manner of the embodiment, the obtaining of the business element information of the plurality of business dimensions associated with the log event comprises:

[0014] At the beginning of a transaction, initializing a thread local storage mapping diagnostic context, and setting the transaction type, transaction name, institution identifier and teller identifier into the mapping diagnostic context;

[0015] When the log event is generated, determining a log category according to its generation position, the log category including bank transaction related logs, communication related logs, thread pool transaction processing logs, database access related logs and background daemon related logs;

[0016] Based on the log category, selecting a corresponding extraction strategy to extract the business element information from a log marker object carried by the log event, the log marker object being associated with the business element information through a preset parent-child relationship.

[0017] As a possible implementation manner of the embodiment, the step S2 comprises the following steps:

[0018] obtaining a THREAD_USAGE_FLAG identifier from the thread-local stored mapping diagnosis context, the identifier representing a log processing scenario of the current thread;

[0019] matching the identifier with a rule key in the single-key dynamic rule set;

[0020] when the identifier matches the rule key in the single-key dynamic rule set successfully, obtaining a corresponding rule value, the rule value containing a log threshold and a decision policy;

[0021] comparing a log level of the log event with the log threshold;

[0022] when the log level is greater than or equal to the log threshold, generating a first filtering decision based on the matched decision policy;

[0023] when the log level is less than the log threshold, generating a first filtering decision based on the unmatched decision policy.

[0024] As a possible implementation manner of the embodiment, the step S3 comprises the following steps:

[0025] matching the service element information with a rule key in the multi-key dynamic rule library, the rule key being a string containing multiple parts, each part corresponding to a service dimension;

[0026] when the service element information matches all parts of the rule key successfully, considering the matching successful, obtaining a corresponding rule value;

[0027] comparing a log level of the log event with a log threshold contained in the rule value to determine whether the matching is successful or not;

[0028] when the matching is successful and the log level of the log event is greater than or equal to the log threshold in the rule value, generating an ACCEPT decision;

[0029] when the matching is successful and the log level of the log event is less than the log threshold in the rule value, generating a DENY decision;

[0030] when the matching fails, generating a decision based on a preset default policy, the default policy including ACCEPT, DENY or NEUTRAL.

[0031] As a possible implementation manner of the embodiment, each part of the rule key supports multiple matching modes, including:

[0032] Direct equality matching mode, requiring that the business element is completely equal to the string in the corresponding part of the rule key;

[0033] ANT style matching mode, supporting wildcard matching;

[0034] Log tag parent-child matching mode, based on the inheritance relationship of the log tag object to perform matching.

[0035] As a possible implementation manner of the embodiment, the step S4 includes the following steps:

[0036] When the first filtering decision is ACCEPT, determining that the final filtering decision is ACCEPT, and directly outputting the log event;

[0037] When the first filtering decision is DENY, determining that the final filtering decision is DENY, and discarding the log event;

[0038] When the first filtering decision is NEUTRAL, taking the second filtering decision as the final filtering decision; wherein,

[0039] If the second filtering decision is ACCEPT, outputting the log event;

[0040] If the second filtering decision is DENY, discarding the log event;

[0041] If the second filtering decision is NEUTRAL, processing according to a preset default final strategy.

[0042] As a possible implementation manner of the embodiment, the step S5 includes the following steps:

[0043] Reading the clearing switch configuration in the application context;

[0044] When the clearing switch configuration indicates the thread clearing mode, generating a file path containing a thread identifier;

[0045] When the clearing switch configuration indicates the teller clearing mode, generating a file path containing an agency identifier and a teller identifier.

[0046] As a possible implementation manner of the embodiment, the generating of the file path containing the thread identifier specifically includes:

[0047] Generating the file path using the format "T${event:ThreadId:-00}", where ThreadId is the thread identifier;

[0048] Outputting the log events of the same thread into the log files under the same path.

[0049] In a second aspect, an embodiment of the present application provides a system log multi-dimensional dynamic filtering and clearing control device, comprising:

[0050] A business element extraction module is configured to receive a log event, acquire business element information of a plurality of business dimensions associated with the log event and a thread context identifier, and the business element information comprises a transaction type, a transaction name, an institution identifier and a teller identifier.

[0051] A single-key dynamic threshold filtering module is configured to input the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, match a preset single-key dynamic rule set, and generate a first filtering decision according to a matching result.

[0052] A multi-key dynamic threshold filtering module is configured to input the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, match a preset multi-key dynamic rule set, and generate a second filtering decision according to a matching result.

[0053] A decision coordination module is configured to determine a final filtering decision of the log event based on a cooperative processing result of the first filtering decision and the second filtering decision.

[0054] A multi-dimensional log dynamic clearing module is configured to dynamically determine an output file path of the log event based on a preset multi-dimensional log dynamic clearing strategy and according to attribute characteristics and environment configurations of the log event.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine readable instructions to perform the steps of any of the above system log multi-dimensional dynamic filtering and clearing control methods.

[0056] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of any of the above system log multi-dimensional dynamic filtering and clearing control methods.

[0057] One of the above technical solutions has the following advantages or beneficial effects:

[0058] 1) The multi-business dimension fine log filtering is realized: the transaction type, the transaction name, the institution identifier, the teller identifier and a plurality of business element dimensions are included in the log filtering decision, the log output control granularity is refined from the module level to the specific business instance level, and the precise log output control can be performed for the specific business scene.

[0059] 2) Support the runtime dynamic update of filtering rules: by supporting the dynamic update of the rule base during the application running, the hot configuration ability of the log output strategy is realized, so that the log detail level does not need to be restarted when adjusting in the production or test environment, and the flexibility and efficiency of operation and maintenance are improved.

[0060] 3) The decision accuracy and efficiency are improved through the cooperation of multi-level filters: the architecture of the single-key filter based on the processing context and the multi-key filter based on the business elements cooperates, combined with the preset decision priority and cooperation strategy, which can realize more accurate and efficient log filtering decisions in complex business scenarios.

[0061] 4) Provide a dynamically adjustable log clearing strategy: through the multi-dimensional log dynamic clearing module, the output path and storage method of the log file can be dynamically selected according to the configuration according to different dimensions such as threads and tellers, thereby improving the retrieval and analysis efficiency of the log file.

[0062] 5) Can effectively reduce invalid log output and improve troubleshooting efficiency: through the above multi-dimensional and dynamic filtering and clearing control, the output amount of irrelevant logs can be significantly reduced while meeting the problem troubleshooting needs, and the development and operation and maintenance personnel can quickly locate the problem, thereby improving the overall efficiency of system operation and maintenance.

[0063] The present application realizes multi-dimensional, dynamic and fine filtering and clearing of business system logs, realizes accurate control and efficient retrieval of log output, improves the efficiency and accuracy of log management, reduces the time cost of problem troubleshooting, and improves the system operation efficiency and problem positioning speed. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 It is a system log multi-dimensional dynamic filtering and clearing control method flow chart according to an exemplary embodiment;

[0065] Figure 2 It is a system log multi-dimensional dynamic filtering and clearing control device structure schematic diagram according to an exemplary embodiment;

[0066] Figure 3 It is a first filtering decision generation flow chart according to an exemplary embodiment;

[0067] Figure 4 It is a second filtering decision generation flow chart according to an exemplary embodiment;

[0068] Figure 5 It is a thread log context matching schematic diagram according to an exemplary embodiment;

[0069] Figure 6 Figure 1 is a multi-filter matching logic flow chart according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] In order to make the technical features of the present application clearer, the present application will be described in detail below with reference to specific embodiments and drawings.

[0071] As shown in Figure 1 Figure 1, the method for multi-dimensional dynamic filtering and clearing control of system logs provided by the embodiment of the present application includes the following steps:

[0072] Step S1, receiving a log event, obtaining business element information of a plurality of business dimensions associated with the log event and a thread context identifier, the business element information including transaction type, transaction name, institution identifier and teller identifier;

[0073] Step S2, inputting the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, matching with a preset single-key dynamic rule set, and generating a first filtering decision according to the matching result;

[0074] Step S3, inputting the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, matching with a preset multi-key dynamic rule library, and generating a second filtering decision according to the matching result;

[0075] Step S4, determining a final filtering decision of the log event based on the cooperative processing result of the first filtering decision and the second filtering decision;

[0076] Step S5, dynamically determining an output file path of the log event according to its attribute characteristics and environmental configuration based on a preset multi-dimensional log dynamic clearing strategy.

[0077] As a possible implementation manner of the embodiment, in step S1, the obtaining of the business element information of the plurality of business dimensions associated with the log event includes:

[0078] At the beginning of a transaction, initializing a thread-local stored mapping diagnostic context, setting the transaction type, transaction name, institution identifier and teller identifier into the mapping diagnostic context;

[0079] At the time when the log event is generated, determining a log category according to its generation position, the log category including bank transaction related logs, communication related logs, thread pool transaction processing logs, database access related logs and background daemon related logs;

[0080] Based on the log category, a corresponding extraction strategy is selected to extract the service element information from a log mark object carried by the log event, the log mark object being associated with the service element information through a preset parent-child relationship.

[0081] As a possible implementation manner of the embodiment, the step S2 comprises the following steps:

[0082] An identifier of THREAD_USAGE_FLAG is obtained from a thread locally stored mapping diagnosis context, the identifier representing a log processing scenario of a current thread;

[0083] The identifier is matched with a rule key in a single key dynamic rule set, the dynamic rule set supporting runtime dynamic updating, comprising: receiving a rule updating request through a management interface or an application programming interface; updating the rule key and a rule value in the dynamic rule set without restarting an application program;

[0084] When the identifier is successfully matched with the rule key in the single key dynamic rule set, a corresponding rule value is obtained, the rule value containing a log threshold and a decision strategy;

[0085] A log level of the log event is compared with the log threshold;

[0086] When the log level is greater than or equal to the log threshold, a first filtering decision is generated based on the successfully matched decision strategy;

[0087] When the log level is less than the log threshold, a first filtering decision is generated based on the unsuccessfully matched decision strategy.

[0088] The value of the identifier of THREAD_USAGE_FLAG comprises:

[0089] DMN, representing a main thread log processing scenario;

[0090] TRC, representing a full link tracking log processing scenario;

[0091] MSG, representing an asynchronous message processing log scenario;

[0092] CSL, representing a control port log scenario;

[0093] CSL_COMM, representing a control port communication log scenario.

[0094] As a possible implementation manner of the embodiment, the step S3 comprises the following steps:

[0095] matching the service element information with a rule key in the multi-key dynamic rule base, the rule key being a string containing multiple parts, each part corresponding to a service dimension;

[0096] when the service element information matches all parts of the rule key successfully, it is considered as a match success, and a corresponding rule value is obtained;

[0097] based on the log threshold contained in the rule value, comparing with the log level of the log event to determine whether the match is successful or not;

[0098] when the match is successful and the log level of the log event is greater than or equal to the log threshold in the rule value, an ACCEPT decision is generated;

[0099] when the match is successful and the log level of the log event is less than the log threshold in the rule value, a DENY decision is generated;

[0100] when the match fails, a decision is generated based on a preset default policy, the default policy including ACCEPT, DENY or NEUTRAL.

[0101] As a possible implementation manner of the embodiment, each part of the rule key supports multiple matching modes, including:

[0102] directly equal matching mode, requiring the string of the service element and the corresponding part of the rule key to be completely equal;

[0103] ANT style matching mode, supporting wildcard matching;

[0104] log tag parent-child matching mode, based on the inheritance relationship of the log tag object for matching.

[0105] As a possible implementation manner of the embodiment, the step S3 further includes a pre-computation caching step:

[0106] when processing a log event for a specific service element combination for the first time, the filtering decision result of each log level under the service element combination is calculated and cached;

[0107] when processing a log event of the same service element combination subsequently, the filtering decision result is directly read from the cache as the second filtering decision.

[0108] The pre-computation caching step further includes cache management:

[0109] setting an access time-based validity period for cache entries;

[0110] setting a maximum capacity limit for the cache, and when the capacity limit is reached, removing cache entries according to the least recently used strategy.

[0111] As a possible implementation manner of the embodiment, the step S4 comprises the following steps:

[0112] When the first filtering decision is ACCEPT, determining the final filtering decision as ACCEPT, and directly outputting the log event;

[0113] When the first filtering decision is DENY, determining the final filtering decision as DENY, and discarding the log event;

[0114] When the first filtering decision is NEUTRAL, taking the second filtering decision as the final filtering decision; wherein,

[0115] If the second filtering decision is ACCEPT, outputting the log event;

[0116] If the second filtering decision is DENY, discarding the log event;

[0117] If the second filtering decision is NEUTRAL, processing according to a preset default final policy.

[0118] As a possible implementation manner of the embodiment, the step S4 further comprises decision priority setting:

[0119] Setting a first priority for the single-key dynamic threshold filter;

[0120] Setting a second priority for the multi-key dynamic threshold filter, the second priority being lower than the first priority;

[0121] Processing the first filtering decision and the second filtering decision according to the priority order.

[0122] As a possible implementation manner of the embodiment, the step S4 further comprises decision coordination policy configuration:

[0123] Providing multiple decision coordination policies, including a business element priority policy, a context identifier priority policy and a weighted decision policy;

[0124] According to system configuration or environmental characteristics, selecting a corresponding decision coordination policy.

[0125] As a possible implementation manner of the embodiment, the step S5 comprises the following steps:

[0126] Reading a clearing switch configuration in an application context;

[0127] When the log splitting switch configuration indicates a per-thread log splitting mode, a file path containing a thread identifier is generated, and the file path containing the thread identifier is generated specifically by using a format "T${event:ThreadId:-00}", where ThreadId is the thread identifier; log events of the same thread are output to log files under the same path;

[0128] When the log splitting switch configuration indicates a per-teller log splitting mode, a file path containing a bank identifier and a teller identifier is generated, and the file path containing the bank identifier and the teller identifier is generated specifically by using a format "${ctx:bankId:-ukn}_${ctx:tellerId:-ukn}", where bankId is the bank identifier and tellerId is the teller identifier; log events of the same bank and the same teller are output to log files under the same path.

[0129] As a possible implementation manner of the embodiment, the step S5 further includes that the environment-aware console outputs:

[0130] detecting a current running environment identifier;

[0131] when the running environment identifier is a development environment or a test environment, automatically enabling console log output;

[0132] when the running environment identifier is a production environment, disabling the console log output.

[0133] As shown in Figure 2 The system log multi-dimensional dynamic filtering and splitting control apparatus provided by the embodiment of the application includes:

[0134] a business element extraction module, configured to receive a log event, acquire business element information of a plurality of business dimensions associated with the log event and a thread context identifier, and the business element information includes a transaction type, a transaction name, a bank identifier and a teller identifier;

[0135] a single-key dynamic threshold filtering module, configured to input the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, match a preset single-key dynamic rule set, and generate a first filtering decision according to a matching result;

[0136] a multi-key dynamic threshold filtering module, configured to input the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, match a preset multi-key dynamic rule library, and generate a second filtering decision according to a matching result;

[0137] A decision coordination module is configured to determine a final filtering decision of the log event based on a result of a coordination process of the first filtering decision and the second filtering decision.

[0138] A multi-dimensional log dynamic distribution module is configured to dynamically determine an output file path of the log event based on a preset multi-dimensional log dynamic distribution strategy and according to attribute characteristics and environment configurations of the log event.

[0139] The application realizes reasonable output and dynamic control of the core system log of the commercial bank by implementing the multi-level dynamic multi-dimensional log filter and the multi-dimensional log dynamic distribution module, improves the flexibility of the log output of each environment, and greatly improves the efficiency and accuracy of the problem troubleshooting of the development and test personnel.

[0140] The specific implementation process of the multi-dimensional dynamic filtering and distribution control of the core system log of the commercial bank mainly involves the following aspects.

[0141] I. Multi-dimensional log dynamic filtering module:

[0142] By configuring the custom filter of log4j2, different filtering levels can be configured for log output according to log categories and business elements. The log category refers to the log classification of different scenarios in the service software code, including bank transaction related, communication related, thread pool transaction processing related, database access related and background daemon related logs. The transaction related log is the key data in the core system of the commercial bank to record the whole life cycle operation of the financial transaction, covering the whole link information from transaction initiation, processing to result return, which has strong business correlation. The communication related log refers to the record of data transmission and information exchange between different components, services or systems. The thread pool transaction processing log refers to the record of thread pool configuration and business processing logic. The database access related log refers to the transaction, connection pool, SQL database access and other function records used to process database access in the transaction process. The background daemon module refers to the record of system startup and stop, connection pool, application index, nacos configuration and other information.

[0143] The multi-dimensional log filtering module is divided into two parts: single key dynamic threshold filter and multi-key dynamic threshold filter.

[0144] 1. Single key dynamic threshold filter:

[0145] The single Key dynamic threshold filter module provides a dynamic log level regulation mechanism based on log context metadata by implementing a custom log filter named CustomSingleKeyDynamicThresholdFilter. The core function of this filter is to intelligently adjust the log output threshold in different business scenarios at runtime according to the preset dynamic identifier in the thread MDC (Mapped Diagnostic Context) combined with the Logger name feature. Its working principle can be summarized as three key stages, as shown in Figure 3

[0146] During program execution, the system dynamically maintains an identifier named THREAD_USAGE_FLAG in the MDC context of each thread. This identifier serves as a key basis for log classification, and its typical values include DMN (main thread log), TRC (full link tracking log), MSG (asynchronous message processing), CSL (control port log), and CSL_COMM (control port communication log) business scenario identifiers. The core processing flow of the filter first retrieves the value of this identifier for the current thread and matches it with the preconfigured key-value mapping rule set (KeyValuePair). When a corresponding rule is successfully matched, the system compares the level of the current log event with the preset threshold level in the rule: if the log level is equal to or higher than the preset threshold (for example, a DEBUG log matches the INFO threshold), it is considered a successful match; otherwise, if it is lower than the threshold, it is determined to be a failed match.

[0147] According to the matching result, the system executes the predefined decision strategy: when configured as ACCEPT, the log event is immediately output and the subsequent processing flow is terminated; when configured as DENY, the log event is discarded directly. It is worth noting that the key-value mapping rule set KeyValuePair supports runtime dynamic updating, which enables operations and maintenance personnel to adjust the log output level in real time without restarting the application, for example, temporarily switching the log level of a specific transaction from INFO to DEBUG when troubleshooting production environment problems.

[0148] ​At the configuration level, the module provides a number of fine-grained control parameters: defaultOnMatch and defaultOnMismatch define the default decision policy when a match is successful and failed, respectively, where ACCEPT means allowing output, and DENY means rejecting the record. The loggerConfigSet parameter is used to limit the effective range of the filter, and when the configuration is empty, it will act on all Logger objects. The noctxDefaultThreshold parameter is specifically designed to handle scenarios where the THREAD_USAGE_FLAG flag is missing in the thread context, and it defines the level threshold and corresponding decision through a comma-separated format. The nommkDefaultThreshold parameter is responsible for handling the case where there is no matching key value in the rule set. When not explicitly configured, both of these fault tolerance mechanisms return a NEUTRAL decision to ensure the continuity of the log processing flow. This hierarchical decision-making mechanism not only guarantees the reachability of core business logs, but also provides flexible technical support for dynamic log governance.

[0149] 2. Multi-key dynamic threshold filter:

[0150] This module defines a custom log filter called CustomMultiKeysDynamicThresholdFilter. At the beginning of a transaction, it initializes business element information such as stxCode (transaction type), txnName (transaction name), branchId (institution id), tellerId (teller id), etc. in the locally stored thread MDC. Business element information can also be attached to the Marker when logging events. Marker refers to the log marker object (org.apache.logging.log4j.Marker) in Log4j2 (Log4j2 is a reliable, fast, and scalable Java logging framework), which is a mechanism for advanced classification and labeling of log events. According to the business element information, rule matching is performed in the pre-configured rule library, as shown in Figure 4

[0151] ​If the business element information matches the rule in the rule base, the corresponding log threshold is used. After successfully matching the corresponding rule, the system compares the level of the current log event with the threshold level preset in the rule: if the log level is equal to or higher than the preset threshold (for example, a DEBUG log matches an INFO threshold), it is considered a successful match; otherwise, if it is lower than the threshold, it is determined to be a failed match. The results of successful or failed matching can be ACCEPT (directly output the log), DENY (discard the log event), and NEUTRAL (pass the log event to the next level of filter). The rules in the rule base can be dynamically adjusted at runtime through the console, thereby achieving dynamic control of log output, facilitating technical testing and troubleshooting.

[0152] The rule base is a plurality of KeyValuePair, wherein the key is a string designed to support multiple matching styles for matching business element information, and the string is divided into four parts separated by commas, which are used to match the transaction type, transaction name, institution id, and clerk id. Each part supports three matching modes: EQL-directly equal, ANT-ANT style matching, and MKR-log Marker parent-child matching, which are concatenated by the matching text and the matching mode with @. If the matching mode is empty, it is defaulted to EQL. The matching step is to first extract the transaction type, transaction name, institution id, and clerk id business elements in the MDC context, and then traverse the KeyValuePair in the rule base to match the four business elements with the four parts of the key string. Only when all four business elements are successfully matched is the match considered successful, as shown in Figure 5 .

[0153] For example, “TCCB, LN03001 @ANT, 806 @ANT, 806777777A01 @EQL” indicates matching transactions with the transaction type TCCB, the transaction name starting with LN03001, the institution id starting with LN03001, and the clerk number starting with 806. Some items can be empty, indicating that whether they meet the item is not screened, such as “TCCB, LN03001 @ANT, 806 @ANT,” which indicates matching transactions with the transaction type TCCB, the transaction name starting with LN03001, and the institution id starting with LN03001.

[0154] The value in KeyValuePair is a string composed of the log threshold, the matching success strategy, and the matching failure strategy, such as value=”info, ACCEPT, DENY”, which indicates that the log event has a log threshold of info at this filter, and if the level is greater than or equal to info, the match is successful and the log is directly output, otherwise, if the level is less than info, the log event is discarded.

[0155] The module also has a pre-computation option, if enabled, the logger will cache the result of whether to output information of each level when the first log event occurs, and then when the next log event occurs, it can quickly judge according to the cached result. This will save the time of matching calculation in the transaction thread when the next log event occurs.

[0156] The log filtering module provides a series of fine-grained configuration parameters for comprehensive control of log filtering behavior. Among them, defaultOnMatch and defaultOnMismatch define the default decision-making strategy when the rule matches successfully or fails, respectively. These two parameters provide a basic behavior criterion for all rules. The loggerConfigSet parameter limits the effective range of the filter. When a specific LoggerConfig name is configured, only the associated Logger object is processed; if it remains empty, it means that the filter will act on all Logger objects, achieving global coverage. For the case of missing context values, the noctxDefaultThreshold parameter provides three-level configuration capability: through a comma-separated string, the default log level threshold, match success decision, and match failure decision can be set at the same time. If this parameter is not configured, when the log context cannot obtain the specified key value, the filter will remain neutral (NEUTRAL), avoiding interference with the log process. Similarly, the nommkDefaultThreshold parameter handles the case of rule library matching failure, also using a three-part configuration structure of "level threshold, match decision, mismatch decision" to ensure that when there is no corresponding match item in the rule library, the system can still execute the preset security policy. The matchLevelCacheSeconds parameter sets the validity period of the cache entry (based on access time counting), supporting -1 (permanent), 0 (disable cache), or any positive integer value (unit: seconds). At the same time, the matchLevelCacheMaxSize parameter limits the maximum number of cache entries. When set to 0, the cache is completely disabled, and a positive value sets a reasonable upper limit according to the system resource status. The cooperative work of these two cache parameters establishes a precise balance between memory occupation and matching efficiency, especially suitable for high-throughput scenarios such as financial trading systems.

[0157] 3. Multi-filter matching logic:

[0158] For example, Figure 6As shown, in the transaction thread, whether the log event output is determined according to the following logic: first, the event is matched by the single-key dynamic threshold filter, if the log category meets the rule screening, and the rule matching result is ACCEPT, it is directly output, the rule matching result is DENY, the log event is discarded, the rule matching result is NEUTRAL, the log event is handed over to the multi-key dynamic threshold filter for judgment, the business elements are extracted for rule matching, if the matching result of the filter is ACCEPT, the log is output, if the matching result is DENY, the log event is discarded. Through the design of the multi-level filter, the log output strategy can be dynamically adjusted according to multiple dimensions of business elements and log categories, greatly improving the flexibility of log output, facilitating bank developers to troubleshoot problems and conduct technical tests.

[0159] Two, multi-dimensional log dynamic sorting module:

[0160] In order to cope with the different log output needs of each environment of the core system of commercial banks, a log system that can conveniently distinguish different log types and dynamically and flexibly adjust the output mode is needed. For example, in the test environment, logs can be temporarily output according to different tellers as needed, so as to facilitate the query of the transaction processing of a certain teller or the output according to different threads to track the track information of the transaction.

[0161] For different log types, the multi-dimensional log dynamic sorting module defines different logger objects, configures different appenders for different logger objects, and defines different output paths, log formats, character sets, rolling strategies, and idle strategies; thereby realizing the differentiation of different log types, which is helpful for accurate log query.

[0162] For the output mode, the multi-dimensional log dynamic sorting module implements a custom property arbiter named CustomPropertyArbiter, which dynamically decides the output mode according to the separate-detail-switch in the spring context. The custom property arbiter can decide whether to enable certain log components according to the configuration property value, and the trcAppender, txnAppender, and glbAppender can support log output according to teller sorting and thread sorting files. The module defaults to thread sorting files, and the logs of different threads are output to different paths. The output mode is "T$$ {event:ThreadId:-00}", so that the log output of a transaction will be output continuously in the text, and the trace of the transaction can be clearly analyzed. In the test environment, the spring context variable configuration can be used to enable the teller sorting file, and the output mode is "$${ctx:bankId:-ukn}_$${ctx:tellerId:-ukn}", which can build a "agency-teller" two-dimensional path, and the log of each teller is stored independently. In this way, when the developer needs to track the operation of a specific teller, the teller's log file can be directly viewed without filtering in the massive logs. This greatly improves the efficiency of problem positioning.

[0163] In addition, a console appender with a property arbiter is added to all loggers, which is enabled when the environment identifier RUN_ENV_FLAG is DEV, DEVC, or other development environments, so that in the development environment, all logs will be output to the console in addition to being output to the file.

[0164] The present application significantly improves the efficiency of log management of the core system of a commercial bank by implementing a multi-dimensional log dynamic filtering and multi-dimensional log dynamic sorting module. Specifically, dynamic fine control of log output in the dimensions of legal person bank, teller, transaction type, transaction name, and other business elements is achieved, and flexible configuration rules are supported at runtime, completely solving the drawbacks of the lack of fine-grained and dynamic control capability in traditional log systems. The innovative dynamic sorting module allows log files to be separated according to the needs of the environment (such as test environment) in the dimensions of teller or thread, enabling development and test personnel to accurately and quickly locate specific teller operations or complete transaction links, greatly improving problem troubleshooting efficiency and accuracy, and reducing the overall time for developers to find problem logs by 57.3%. In addition, the environment-aware output strategy (such as automatic output to the console in the development environment) further optimizes the development and debugging experience. Overall, the present application provides a more flexible, efficient, and accurate log output and control scheme for commercial banks, effectively supporting system operation and maintenance, problem diagnosis, and performance optimization.

[0165] An electronic device provided by an embodiment of the present application comprises a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the processor executes the machine readable instructions to perform steps of the method for multi-dimension dynamic filtering and clearing control of system log as described above.

[0166] Specifically, the memory and the processor can be general memory and processor, which are not specifically limited here, and when the processor runs the computer program stored in the memory, the method for multi-dimension dynamic filtering and clearing control of system log as described above can be executed.

[0167] Corresponding to the starting method of the application program, an embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform steps of the method for multi-dimension dynamic filtering and clearing control of system log as described above.

[0168] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application rather than limit it, although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by the equivalent, without departing from the spirit and scope of the present application, any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A method for multi-dimension dynamic filtering and clearing control of system logs, characterized in that, The method comprises the following steps: Step S1, receiving a log event, obtaining business element information of a plurality of business dimensions associated with the log event, and a thread context identifier, the business element information comprising a transaction type, a transaction name, an institution identifier, and a teller identifier; Step S2, inputting the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, matching a preset single-key dynamic rule set, and generating a first filtering decision according to a matching result; Step S3, inputting the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, matching a preset multi-key dynamic rule library, and generating a second filtering decision according to a matching result; Step S4, determining a final filtering decision of the log event based on a cooperative processing result of the first filtering decision and the second filtering decision; Step S5, dynamically determining an output file path of the log event based on a preset multi-dimensional log dynamic sorting strategy and according to attribute characteristics and environment configurations of the log event; The step S3 comprises the following steps: matching the business element information with a rule key in the multi-key dynamic rule library, the rule key being a string comprising a plurality of parts, each part corresponding to a business dimension; when the business element information matches all parts of the rule key successfully, it is considered that the matching is successful, and a corresponding rule value is obtained; comparing a log threshold value contained in the rule value with a log level of the log event to determine whether the matching is successful or not; when the matching is successful and the log level of the log event is greater than or equal to the log threshold value in the rule value, an ACCEPT decision is generated; when the matching is successful and the log level of the log event is less than the log threshold value in the rule value, a DENY decision is generated; when the matching fails, a decision is generated based on a preset default strategy, the default strategy comprising ACCEPT, DENY, or NEUTRAL; each part of the rule key supports a plurality of matching modes, comprising: a direct equality matching mode, requiring that a business element and a corresponding part of a rule key string are completely equal; an ANT style matching mode, supporting wildcard matching; a log marker parent-child matching mode, matching based on an inheritance relationship of a log marker object.

2. The method for system log multi-dimension dynamic filtering and clearing control according to claim 1, characterized in that, The obtaining of the business element information of the plurality of business dimensions associated with the log event comprises: initializing a thread local storage mapping diagnostic context at the beginning of a transaction, and setting the transaction type, the transaction name, the institution identifier, and the teller identifier into the mapping diagnostic context; determining a log category according to a generation position of the log event when the log event is generated, the log category comprising a bank transaction related log, a communication related log, a thread pool transaction processing log, a database access related log, and a background daemon related log; based on the log category, selecting a corresponding extraction strategy to extract the business element information from a log marker object carried by the log event, the log marker object being associated with the business element information through a preset parent-child relationship.

3. The method for system log multi-dimension dynamic filtering and clearing control according to claim 1, characterized in that, The step S2 comprises the following steps: obtaining a THREAD_USAGE_FLAG identifier from a mapping diagnosis context MDC stored locally by a thread; matching the THREAD_USAGE_FLAG identifier with a rule key in the single-key dynamic rule set; when the matching is successful, obtaining a corresponding rule value, the rule value including a log threshold and a decision strategy; comparing a log level of the log event with the log threshold, and generating a first filtering decision based on a comparison result and the decision strategy.

4. The method for system log multi-dimension dynamic filtering and clearing control according to claim 1, characterized in that, The step S4 includes the following steps: when the first filtering decision is ACCEPT, determining that the final filtering decision is ACCEPT, and directly outputting the log event; when the first filtering decision is DENY, determining that the final filtering decision is DENY, and discarding the log event; when the first filtering decision is NEUTRAL, taking the second filtering decision as the final filtering decision; wherein, if the second filtering decision is ACCEPT, outputting the log event; if the second filtering decision is DENY, discarding the log event; if the second filtering decision is NEUTRAL, processing according to a preset default final strategy.

5. The method for system log multi-dimension dynamic filtering and clearing control according to any one of claims 1-4, characterized in that, The step S5 includes the following steps: reading a clearing switch configuration in an application context; when the clearing switch configuration indicates a thread clearing mode, generating a file path including a thread identifier; when the clearing switch configuration indicates a teller clearing mode, generating a file path including an agency identifier and a teller identifier.

6. A system log multi-dimension dynamic filtering and clearing control device, characterized in that, The apparatus for implementing the method for multi-dimensional dynamic filtering and clearing control of system logs according to any one of claims 1-5, the apparatus comprising: a business element extraction module configured to receive a log event, obtain business element information of a plurality of business dimensions associated with the log event, and obtain a thread context identifier, the business element information including a transaction type, a transaction name, an agency identifier, and a teller identifier; a single-key dynamic threshold filtering module configured to input the thread context identifier associated with the log event into a preset single-key dynamic threshold filter, match a preset single-key dynamic rule set, and generate a first filtering decision according to a matching result; a multi-key dynamic threshold filtering module configured to input the business element information of the plurality of business dimensions into a preset multi-key dynamic threshold filter, match a preset multi-key dynamic rule library, and generate a second filtering decision according to a matching result; a decision coordination module configured to determine a final filtering decision of the log event based on a coordinated processing result of the first filtering decision and the second filtering decision; a multi-dimensional log dynamic clearing module configured to dynamically determine an output file path of the log event based on a preset multi-dimensional log dynamic clearing strategy and attribute characteristics and environmental configurations of the log event.

7. An electronic device, comprising: The system comprises a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor communicates with the memory through the bus, the processor executes the machine readable instructions to execute the steps of the system log multi-dimension dynamic filtering and clearing control method as claimed in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is run by the processor, the steps of the system log multi-dimension dynamic filtering and clearing control method as claimed in any one of claims 1-5 are executed.

Images