Log processing method and device based on ternary management system

Abstract

The invention provides a log processing method and device based on a ternary management system, and belongs to the technical field of software development.The log processing method based on the ternary management system comprises the steps that a log mode is obtained through a log generation strategy configured for a target request by a strategy engine; whether the logs are stored or not is controlled according to requirements through the intermediate log aggregation system, the logs are finally stored in the target database, a dynamic and structured log management mode is provided, the needed logs can be flexibly obtained without code modification, log recording and monitoring of a large-scale distributed system can be effectively supported, and the log management efficiency is improved. The deployment cost is reduced; and the log management efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software development, and in particular to a log processing method and device based on a ternary management system. Background Art

[0002] Traditional log management usually uses the method of directly embedding log frameworks in the code, such as Log4j and SLF4J, to record runtime information to log files. At the same time, in order to meet the needs of persistent storage, system developers often use filters or interceptor technology to capture specific HTTP requests and store related log information in the database. However, this direct write log management method has obvious drawbacks, especially in complex microservice architectures, where the consistency problem between log files and database records is becoming increasingly prominent. The log configuration is tightly bound to the code, and any change in log requirements means code modification, which increases the difficulty of maintenance. At the same time, every time a new log module is added, the code needs to be modified and redeployed, causing unnecessary development burden. Summary of the invention

[0003] The present invention provides a log processing method and device based on a ternary management system, which are used to solve the defect of relatively complex Japanese modification management in the prior art, and achieve the effect of reducing deployment costs and improving log management efficiency.

[0004] The present invention provides a log processing method based on a ternary management system, comprising: In the case where a target request reaches a target service, a log mode of the target request is determined based on a policy engine integrated in an interceptor corresponding to the target service; the log mode is determined based on a log generation policy configured for the target request by the policy engine; the log mode is a predefined structure after the target request is configured based on the log generation policy; Based on the log mode of the target request, the policy engine sends the log mode of the target request to the intermediate log aggregation system; When the intermediate log aggregation system determines that logs need to be stored, the intermediate log aggregation system sends log acquisition information to the target service; the log acquisition information includes a log template corresponding to the target database; Based on the log mode of the target request, generate a target log through the target service according to the log template corresponding to the target database, and send it to the intermediate log aggregation system; The target log is stored in the target database through the intermediate log aggregation system.

[0005] According to a log processing method based on a ternary management system provided by the present invention, the policy engine integrated in the interceptor corresponding to the target service determines the log mode of the target request, including: Based on the log generation policy configured by the policy engine for the target request, determine a target tag included in the target request, where the target tag is used to represent user information, path information, address information, and tag information corresponding to the target request; Based on the target tag included in the target request, a log mode of the target request is determined.

[0006] According to a log processing method based on a ternary management system provided by the present invention, the log mode of the target request is determined based on a target tag included in the target request, including: Based on the user information of the target tag, the user operation information in the log mode is determined; based on the path information of the target tag, the initial log format in the log mode is determined; based on the address information of the target tag, the security information in the log mode is determined; based on the tag information of the target tag, the identifier in the log mode is determined.

[0007] According to a log processing method based on a ternary management system provided by the present invention, the marking information includes at least one of abnormal behavior pattern information, user feedback marking information and risk marking information.

[0008] According to a log processing method based on a ternary management system provided by the present invention, the intermediate log aggregation system determines that logs need to be stored, including: Receiving, through the intermediate log aggregation system, a log management input of the user for the target request; Based on the log management input, determining a target database for storing logs corresponding to the target request; The intermediate log aggregation system determines that the target database meets the storage requirements and that logs need to be stored.

[0009] According to a log processing method based on a ternary management system provided by the present invention, determining that the target database meets the storage requirements through the intermediate log aggregation system includes: Determining the type storage requirement of the target database through the intermediate log aggregation system; In the case where the log mode meets the type storage requirement of the target database, determining a storage method for the log of the target request; It is determined that the target database satisfies the storage mode.

[0010] According to a log processing method based on a ternary management system provided by the present invention, storing the target log to the target database through the intermediate log aggregation system includes: The target log is cached by the intermediate log aggregation system to respond to the request of the target service for the target log, and the target log is stored in the target database to respond to the request of the user for the target log.

[0011] The present invention also provides a log processing device based on the ternary management system, comprising: A first processing module is used to determine the log mode of the target request based on the policy engine integrated in the interceptor corresponding to the target service when the target request arrives at the target service; the log mode is determined based on the log generation policy configured by the policy engine for the target request; the log mode is a predefined structure after the target request is configured based on the log generation policy; A second processing module, configured to send the log mode of the target request to the intermediate log aggregation system through a policy engine based on the log mode of the target request; A third processing module is used to send log acquisition information to the target service through the intermediate log aggregation system when the intermediate log aggregation system determines that logs need to be stored; the log acquisition information includes a log template corresponding to the target database; A fourth processing module, configured to generate a target log according to a log template corresponding to the target database through the target service based on the log mode of the target request, and send the target log to the intermediate log aggregation system; The fifth processing module is used to store the target log in the target database through the intermediate log aggregation system.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a log processing method based on a ternary management system as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the log processing method based on the ternary management system as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned log processing methods based on the ternary management system.

[0015] The log processing method and device based on the ternary management system provided by the present invention obtain the log mode through the log generation strategy configured by the policy engine for the target request, and control whether to store the log according to the demand through the intermediate log aggregation system, and finally store the log in the target database, providing a dynamic and structured log management method, which can flexibly obtain the required logs without modifying the code, can effectively support the log recording and monitoring of large-scale distributed systems, reduce the deployment cost and improve the log management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the log processing method based on the ternary management system provided by the present invention; Figure 2 This is the second flow chart of the log processing method based on the ternary management system provided by the present invention; Figure 3 It is a structural schematic diagram of a log processing device based on a ternary management system provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Combine the following Figure 1-Figure 4 The present invention describes a log processing method and device based on a ternary management system.

[0020] like Figure 1 As shown, the log processing method based on the ternary management system according to the embodiment of the present invention mainly includes step 110 , step 120 , step 130 , step 140 and step 150 .

[0021] Step 110, when the target request reaches the target service, the log mode of the target request is determined based on the policy engine integrated in the interceptor corresponding to the target service.

[0022] The log mode is determined based on the log generation policy configured by the policy engine for the target request; the log mode is a predefined structure after the target request is configured based on the log generation policy.

[0023] When a user initiates a request and reaches the target service, the request is first processed by the target service's interceptor. The interceptor is an intermediate layer that intercepts requests and responses and can perform specific operations before or after the target request is processed.

[0024] The interceptor integrates a policy engine, which is a configurable logic component that dynamically determines the logging method and content based on different request characteristics such as user identity, request type, path information, timestamp, etc. The policy engine determines the mode of generating logs by parsing the relevant information of the target request.

[0025] The policy engine uses pre-defined log generation policies to analyze different attributes of the target request, such as path, method, user identity, etc., to determine what information needs to be recorded and how to format the information.

[0026] The log mode is a structural template based on these policy definitions, which predefines the log content, format, field, level, etc. For example, the log mode may specify that the request time, user ID, request path and other information be recorded in JSON format, or adopt other log structures.

[0027] like Figure 2 As shown, based on the policy engine integrated in the interceptor corresponding to the target service, the log mode of the target request is determined, including step 111 and step 112.

[0028] Step 111 , based on the log generation policy configured by the policy engine for the target request, determine the target tag included in the target request.

[0029] Step 112: Determine a log mode of the target request based on the target tag included in the target request.

[0030] The target tag is used to indicate the user information, path information, address information, and tag information corresponding to the target request.

[0031] The log generation policy is a set of rules configured by the system administrator or developer that specifies how to record key information in the request. The policy generation logic determines which information needs to be recorded as a log based on different characteristics of the request, such as the request path, request method, user identity, etc.

[0032] The policy engine makes decisions based on these configurations. It may select different tags based on different request types such as API requests, page requests, etc.

[0033] The tags of user information may include user ID, user name, user role, etc. It can be extracted from the request header such as authorization information or the request body such as the username field.

[0034] The path information tag may include the requested URL path, request method (such as GET, POST, etc.), query parameters, etc. It is usually obtained by parsing the URL or path field of the HTTP request.

[0035] The tag of the address information may include the source IP address of the request, client device information, geographic location, etc. It can be obtained through the X-Forwarded-For, RemoteAddr field or other relevant information in the request header.

[0036] The tag of the marking information can contain additional marking data, such as whether the request requires special processing, whether it is a test request, etc. This information is usually generated dynamically by the application or passed to the policy engine.

[0037] Based on the configured log policy, the policy engine dynamically parses the target request, extracts the above key fields, and generates the corresponding target tag. For example, the policy engine may extract this information by accessing various parts of the request such as the request header, request body, query parameters, etc., and map it to the target tag.

[0038] In some embodiments, the log mode of the target request is determined based on the target tag included in the target request, including: determining the user operation information in the log mode based on the user information of the target tag; determining the initial log format in the log mode based on the path information of the target tag; determining the security information in the log mode based on the address information of the target tag; and determining the identifier in the log mode based on the tag information of the target tag.

[0039] The user operation information in the log mode is used to describe the user's specific operation on the system. For example, a user may initiate a specific API request or access a specific resource. This part of the log mode can include: user ID or user name; user behavior type such as login, data modification, data query, etc.; specific time and duration of user operation, etc.

[0040] Assume that a user with user ID 12345 initiates a request. The request may contain the user ID field as the target tag, and the log pattern may contain the information "user 12345 performed a certain operation".

[0041] The initial log format refers to the initial organizational structure of the log, which determines what key information is recorded in the log and in what initial format. Path information usually helps determine how the log is organized. For example, if the request path is / api / v1 / users, the log format may include this path as a key field, indicating that the log is a user-related operation. If the request path is a specific resource path, the log format may emphasize the resource name, request method (GET / POST), parameters, etc.

[0042] Assuming the path is / api / v1 / login, the initial format of the log can specify the request method (such as GET / POST), timestamp, and possible user information related to this path.

[0043] Address information usually refers to the IP address, geographic location, client information (such as device type) of the request source. This information is closely related to security. Security information refers to the detailed security information about the source of the request contained in the log. For example, the source IP address may help determine whether it is a malicious request, and the client device information can help identify potential abnormal behavior. Security information can include: the source IP address of the request, the geographic location of the request source (based on IP positioning), the type of device that initiated the request (such as mobile device, PC, etc.), and whether the request comes from a known safe area or network.

[0044] If the source IP of the target request is 192.xxx.1.xxx, the log may record the geographical location, device information, and whether it is a suspicious source.

[0045] The tag information may be any additional data used to identify the characteristics of the request, such as whether it is a batch operation, a background task, an administrator operation, etc. These tags help determine the special nature of the request.

[0046] The marking information includes at least one of abnormal behavior pattern information, user feedback marking information and risk marking information.

[0047] Identifiers are used to mark specific information in logs, usually to help identify the special type or status of a request. For example: whether the request is batch processing, whether the request is a special type of operation (such as administrator operation, test request, etc.), and whether a special processing is required (such as sensitive operation, key data modification, etc.).

[0048] For example, if the tag information contains admin_action, the identifier part in the log can be marked as an administrator action or a sensitive action.

[0049] In the above scenario, a multi-dimensional log mode can be constructed based on the information of the target tag, so that the log record is not just a simple record of the request, but also a detailed analysis of each dimension of the request, thereby providing rich support for subsequent auditing, monitoring, security analysis, etc.

[0050] Step 120 : Based on the log mode of the target request, the log mode of the target request is sent to the intermediate log aggregation system through the policy engine.

[0051] Once the log pattern is determined, the interceptor of the target service will send the log pattern to the intermediate log aggregation system through the policy engine. The intermediate log aggregation system is a centralized and independent log collection platform that is responsible for storing and processing log information from different services.

[0052] The policy engine sends the log pattern to the intermediate log aggregation system, usually in the form of structured data (such as JSON or XML), ensuring that the system can correctly parse and perform subsequent log storage and analysis operations.

[0053] Step 130: When the intermediate log aggregation system determines that the log needs to be stored, the intermediate log aggregation system sends log acquisition information to the target service.

[0054] The log acquisition information includes the log template corresponding to the target database. After receiving the log pattern, the intermediate log aggregation system first checks whether the log needs to be stored. It usually decides whether to save the log of the request based on certain rules or conditions such as the log level, request type, request frequency, etc.

[0055] When deciding to store logs, the intermediate log aggregation system will send log acquisition information to the target service, requiring the target service to generate specific logs according to the log mode. The log acquisition information usually contains relevant information about the target database, log template, field requirements, etc.

[0056] The log template is a predefined log structure in the target database, which usually includes the fields required to store logs in the database, such as the log timestamp, log level, request path, user ID, etc. The intermediate log aggregation system will require the target service to generate logs that meet the database structure requirements based on this template. Step 140 , based on the log mode of the target request, a target log is generated through the target service according to a log template corresponding to the target database, and sent to the intermediate log aggregation system.

[0057] The target service generates specific log data according to the log template received from the intermediate log aggregation system and the log mode.

[0058] Once the target service generates logs, it sends the log data to the intermediate log aggregation system through network protocols such as HTTP, MQ, gRPC, etc. This process is carried out through secure and reliable communication methods to ensure the integrity of the log data and the security of transmission.

[0059] Step 150: Store the target log into the target database through the intermediate log aggregation system.

[0060] After the intermediate log aggregation system receives the logs sent by the target service, it will further process them according to the system configuration and log storage strategy.

[0061] Based on the received log data and the configuration of the target database, the logs can be stored in the target database. The target database is usually a relational database or a distributed database, and the log data is stored by field for subsequent query and analysis.

[0062] During the storage process, some optimization measures can be taken, such as log compression, partitioned storage, data deduplication, etc., to ensure that log data can be stored efficiently and easy to query.

[0063] It can be understood that the ternary management system of the embodiment of the present invention is a system for managing and configuring logs, and it can perform log management through configuration management, log reading and storage management, and maintainability and extensibility management.

[0064] In the present embodiment, the generation, storage, output mode and other settings of log are managed by URL configuration and menu configuration. The mode of this configuration management makes the management of log become more flexible, and developers no longer need to directly modify the code to adjust the configuration related to log, thereby reducing the coupling between code and configuration. Through log reading and storage management, the consistency between the reading mechanism of log file and the database record is effectively solved. The ternary management system can help ensure the synchronization and consistency of log content between the file system and the database, and avoids the problem that the common log file and database record do not match in the past.

[0065] Maintainability and scalability management Through menu-based configuration and centralized log management configuration, the maintainability and scalability of the system have been significantly improved. Since the log configuration is decoupled from the code, future expansion of the system, such as adding new log functions or modifying the log format, becomes easier without the need to frequently change and redeploy the code.

[0066] In this embodiment, the policy engine in the ternary management system is used to finely configure the URL of each system method to ensure that only the log data corresponding to the URL that has been screened by the preset conditions can be collected and finally stored. It is also possible to define a set of standardized log output formats, and all log records will follow this unified structure to facilitate subsequent automated processing and analysis.

[0067] It is understandable that the use of an advanced log processing engine, namely the intermediate log aggregation system, has the ability to read log files in a scheduled and real-time manner, intelligently parse and structure each line of log, and then synchronize the processed log data to the target database according to preset rules to ensure the consistency of the log files and the records of the target database.

[0068] In addition, users can also dynamically add, delete and configure log modules of the intermediate log aggregation system through the system menu without touching the underlying code, which greatly simplifies the complexity of log management.

[0069] According to the log processing method based on the ternary management system provided by the embodiment of the present invention, the log mode is obtained by configuring the log generation policy of the target request through the policy engine, and the intermediate log aggregation system is used to control whether to store the log according to the demand, and finally the log is stored in the target database. A dynamic and structured log management method is provided, and the required logs can be flexibly obtained without modifying the code. It can effectively support the log recording and monitoring of large-scale distributed systems, reduce deployment costs and improve log management efficiency.

[0070] In some embodiments, the intermediate log aggregation system determines that logs need to be stored, including the following process.

[0071] The user's log management input for the target request can be received through the intermediate log aggregation system; based on the log management input, the target database for log storage corresponding to the target request is determined; through the intermediate log aggregation system, it is determined that the target database meets the storage requirements and that the logs need to be stored.

[0072] User input log management input can include query conditions, log level, storage format, storage location, etc. of the requested log. Users send log management instructions to the system through an interface (such as a web interface or API request). The intermediate log aggregation system needs an interface to receive these inputs, which can be API, web form, or message queue. The system will parse these inputs and identify the relevant parameters of the log.

[0073] Based on the log management input, the intermediate log aggregation system needs to know which database to store the logs in. The system can pre-configure some rules or policies to associate specific types of logs with specific databases. For example: error logs are stored in a relational database, high-frequency logs are stored in a NoSQL database, and security logs are stored in a dedicated log database.

[0074] According to the log conditions input by the user, it is necessary to query and determine which database is the most suitable storage location, and then use the configuration file or database table to store the mapping relationship between the log type and the database. Alternatively, the target database can be selected based on user input and rules.

[0075] The target database needs to meet specific storage requirements, such as capacity, performance, data structure, etc., which can be verified in the following ways.

[0076] The database capacity check is used to determine whether the target database has enough space to store log data. The performance requirement is used to check the write performance of the target database to ensure that it can handle the expected log write rate. The compatibility check is used to ensure that the target database can handle the log format (such as JSON, XML, text) or the log query method (such as supporting full-text retrieval, etc.). The storage verification is used to check the health, storage space, and performance of the database through the connection interface with the target database to ensure that it meets the storage requirements.

[0077] You can use the database monitoring tools or API interfaces to check storage status, performance, etc. Design the verification logic of storage requirements in the system and verify it in combination with the actual database indicators.

[0078] Once the target database is determined and verified to meet the storage requirements, the system needs to decide which specific logs need to be stored. This process usually depends on the level, time range, source, etc. of the logs.

[0079] For example, some logs may only need to be stored for a short period of time, or only a certain type of log data may be stored. The filtered log data will be sent to the target database for storage through the intermediate log aggregation system. At this point, the system needs to ensure that the data structure of the log matches the database schema and perform effective storage operations.

[0080] Filter logs based on specific conditions (such as log level or time).

[0081] The filtered logs are stored in the target database through database write interfaces such as SQL INSERT and NoSQL insert operations. The system can use an intermediate layer to coordinate input, storage target selection, storage verification, and log storage operations. This intermediate layer can be a microservice that provides an API to receive user log management input and implement the above process.

[0082] You can use a rule engine or configuration management system to dynamically determine the target database. Automatically select the target database based on the characteristics of the log and the preset strategy. You can also use a database connection pool to handle database operations, and have a database monitoring system to monitor the health of the database to ensure that storage requirements are met.

[0083] To improve performance, the system can use log aggregation and batch processing techniques to reduce frequent database write operations. Logs may be cached first and then written to the target database in batches.

[0084] The key to the intermediate log aggregation system is to receive user input, determine the storage target, and ensure that the target database meets the storage requirements. Through steps such as rule engine, database health check, and storage verification, the system will eventually store the log data in the appropriate database, thereby achieving efficient and reliable log management.

[0085] In some embodiments, determining that a target database meets storage requirements through an intermediate log aggregation system includes: determining the type storage requirements of the target database through an intermediate log aggregation system; determining a storage method for logs of target requests when a log mode meets the type storage requirements of the target database; and determining that the target database meets the storage method.

[0086] You can identify the storage requirements of the target database based on its type, such as relational database, NoSQL database, or log storage system, including data format, structure, and performance requirements.

[0087] The structure of log data usually includes timestamp, log level, message content, etc. Different types of log data such as text logs, JSON logs, XML logs, etc. may not match the requirements of different databases. Therefore, it is necessary to judge whether it is suitable for the target database based on the log mode.

[0088] The structure of log data (such as JSON, text, etc.) can be analyzed to determine whether it meets the storage requirements of the target database, and format conversion can be performed if necessary.

[0089] Based on the target database and log mode, select the appropriate storage method (such as table, document, key-value pair, etc.) and ensure that the query and performance requirements are met. Ensure that the target database can meet the requirements of the selected storage method and verify its performance, capacity, etc.

[0090] If the target database supports structured storage such as a relational database, log data may need to be stored in a table format.

[0091] If the target database supports unstructured storage (such as a NoSQL database), log data can be directly stored as documents, key-value pairs, etc.

[0092] Dedicated logging systems such as Elasticsearch may require storage in a specific index format to support efficient query and retrieval.

[0093] In some embodiments, the target log is stored in the target database through the intermediate log aggregation system, including: caching the target log through the intermediate log aggregation system to respond to the target service's request for the target log, and storing the target log in the target database to respond to the user's request for the target log.

[0094] The intermediate log aggregation system receives log requests from target services such as applications, microservices, etc., or users. The target service may generate logs periodically or according to specific conditions.

[0095] When the target service has a need for request logs, the intermediate log aggregation system will cache these log data in memory (for example, using a memory cache system such as Redis, Memcached, etc.). The cache mechanism can optimize performance by setting timeouts, cache sizes, and expiration policies. In particular, when queries are frequent, results can be returned directly from the cache.

[0096] When the number of target log requests reaches a certain number or the logs in the cache expire, a cache refresh operation can be triggered to store the logs only in the target database.

[0097] Newly received log data can be written to the target database in real time to ensure that data is not lost.

[0098] For high-frequency log generation scenarios, you can periodically write logs in the cache into the database in batches, for example, every minute or every hour.

[0099] Before storing logs, they need to be formatted, converted, or cleaned, such as parsing JSON format, converting timestamps, etc., to meet the storage requirements of the target database.

[0100] When a user requests a log, the intermediate log aggregation system first checks whether the log data exists in the cache. If it exists, it is returned directly from the cache. If the cache does not hit, the system queries the log from the target database.

[0101] In order to speed up the log query process, an index may be set for the log database to optimize the query speed.

[0102] You can also set appropriate cache expiration time and clearing policies to ensure that the data in the cache is not outdated or excessive. For example, use the LRU (least recently used) policy to automatically clear expired or infrequently accessed logs. In order to manage database storage space, you can also archive historical logs regularly or delete them based on certain conditions (such as date, log type, etc.).

[0103] It can be understood that through the intermediate log aggregation system, the target log is first cached to speed up the service response, and the log data is written to the target database in batches or in real time as needed, while providing an effective log query interface to ensure the efficiency and availability of the system.

[0104] The following is a description of a log processing device based on a ternary management system provided by the present invention. The log processing device based on a ternary management system described below and the log processing method based on a ternary management system described above can be referred to in correspondence with each other.

[0105] like Figure 3 As shown, the log processing device based on the ternary management system according to the embodiment of the present invention mainly includes a first processing module 310 , a second processing module 320 , a third processing module 330 , a fourth processing module 340 and a fifth processing module 350 .

[0106] The first processing module 310 is used to determine the log mode of the target request based on the policy engine integrated in the interceptor corresponding to the target service when the target request arrives at the target service; the log mode is determined based on the log generation policy configured by the policy engine for the target request; the log mode is a predefined structure after the target request is configured based on the log generation policy; The second processing module 320 is used to send the log mode of the target request to the intermediate log aggregation system through the policy engine based on the log mode of the target request; The third processing module 330 is used to send log acquisition information to the target service through the intermediate log aggregation system when the intermediate log aggregation system determines that the log needs to be stored; the log acquisition information includes a log template corresponding to the target database; The fourth processing module 340 is used to generate a target log according to a log template corresponding to the target database through a target service based on the log mode of the target request, and send the target log to the intermediate log aggregation system; The fifth processing module 350 is used to store the target log in the target database through the intermediate log aggregation system.

[0107] According to the log processing device based on the ternary management system provided by the embodiment of the present invention, the log mode is obtained by the log generation strategy configured by the policy engine for the target request, and the intermediate log aggregation system is used to control whether to store the log according to the demand, and finally the log is stored in the target database. A dynamic and structured log management method is provided, and the required logs can be flexibly obtained without modifying the code. It can effectively support the log recording and monitoring of large-scale distributed systems, reduce deployment costs and improve log management efficiency.

[0108] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the log processing method based on the ternary management system, which includes: when the target request arrives at the target service, based on the policy engine integrated in the interceptor corresponding to the target service, determine the log mode of the target request; the log mode is determined based on the log generation policy configured for the target request by the policy engine; the log mode is a predefined structure after the target request is configured based on the log generation policy; based on the log mode of the target request, send the log mode of the target request to the intermediate log aggregation system through the policy engine; when the intermediate log aggregation system determines that the log needs to be stored, send log acquisition information to the target service through the intermediate log aggregation system; the log acquisition information includes the log template corresponding to the target database; based on the log mode of the target request, generate the target log according to the log template corresponding to the target database through the target service, and send it to the intermediate log aggregation system; store the target log in the target database through the intermediate log aggregation system.

[0109] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log processing method based on the ternary management system provided by the above methods, the method including: when a target request arrives at a target service, determining the log mode of the target request based on the policy engine integrated in the interceptor corresponding to the target service; the log mode is determined based on the log generation policy configured for the target request by the policy engine; the log mode is a predefined structure after the target request is configured based on the log generation policy; based on the log mode of the target request, sending the log mode of the target request to the intermediate log aggregation system through the policy engine; when the intermediate log aggregation system determines that the log needs to be stored, sending log acquisition information to the target service through the intermediate log aggregation system; the log acquisition information includes the log template corresponding to the target database; based on the log mode of the target request, generating the target log according to the log template corresponding to the target database through the target service and sending it to the intermediate log aggregation system; storing the target log in the target database through the intermediate log aggregation system.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the log processing method based on the ternary management system provided by the above-mentioned methods, the method comprising: when a target request arrives at a target service, determining the log mode of the target request based on a policy engine integrated in an interceptor corresponding to the target service; the log mode is determined based on a log generation policy configured for the target request by the policy engine; the log mode is a predefined structure after the target request is configured based on the log generation policy; based on the log mode of the target request, sending the log mode of the target request to an intermediate log aggregation system through the policy engine; when the intermediate log aggregation system determines that logs need to be stored, sending log acquisition information to the target service through the intermediate log aggregation system; the log acquisition information includes a log template corresponding to the target database; based on the log mode of the target request, generating a target log through the target service according to the log template corresponding to the target database, and sending it to the intermediate log aggregation system; storing the target log in the target database through the intermediate log aggregation system.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log processing method based on a ternary management system, characterized in that: include: In the case where the target request reaches the target service, determining the log mode of the target request based on the policy engine integrated in the interceptor corresponding to the target service; The log mode is determined based on the log generation policy configured by the policy engine for the target request; The log mode is a predefined structure after configuring the target request based on the log generation strategy; Based on the log mode of the target request, the policy engine sends the log mode of the target request to the intermediate log aggregation system; When the intermediate log aggregation system determines that logs need to be stored, sending log acquisition information to the target service through the intermediate log aggregation system; The log acquisition information includes a log template corresponding to the target database; Based on the log mode of the target request, generate a target log through the target service according to the log template corresponding to the target database, and send it to the intermediate log aggregation system; The target log is stored in the target database through the intermediate log aggregation system.

2. The log processing method based on the ternary management system according to claim 1 is characterized in that: The determining, based on a policy engine integrated in an interceptor corresponding to the target service, a log mode of the target request includes: Based on the log generation policy configured by the policy engine for the target request, determine a target tag included in the target request, where the target tag is used to represent user information, path information, address information, and tag information corresponding to the target request; Based on the target tag included in the target request, a log mode of the target request is determined.

3. The log processing method based on the ternary management system according to claim 2 is characterized in that: The determining the log mode of the target request based on the target tag included in the target request includes: Based on the user information of the target tag, the user operation information in the log mode is determined; based on the path information of the target tag, the initial log format in the log mode is determined; based on the address information of the target tag, the security information in the log mode is determined; based on the tag information of the target tag, the identifier in the log mode is determined.

4. The log processing method based on the ternary management system according to claim 2 is characterized in that: The marking information includes at least one of abnormal behavior pattern information, user feedback marking information and risk marking information.

5. The log processing method based on the ternary management system according to claim 1 is characterized in that: The intermediate log aggregation system determines that logs need to be stored, including: Receiving, through the intermediate log aggregation system, a log management input of the user for the target request; Based on the log management input, determining a target database for storing logs corresponding to the target request; The intermediate log aggregation system determines that the target database meets the storage requirements and that logs need to be stored.

6. The log processing method based on the ternary management system according to claim 5 is characterized in that: The determining, by the intermediate log aggregation system, that the target database meets the storage requirement includes: Determining the type storage requirement of the target database through the intermediate log aggregation system; In the case where the log mode meets the type storage requirement of the target database, determining a storage method for the log of the target request; It is determined that the target database satisfies the storage mode.

7. The log processing method based on the ternary management system according to claim 1 is characterized in that: The storing the target log to the target database through the intermediate log aggregation system includes: The target log is cached by the intermediate log aggregation system to respond to the request of the target service for the target log, and the target log is stored in the target database to respond to the request of the user for the target log.

8. A log processing device based on a ternary management system, characterized in that: include: A first processing module is used to determine the log mode of the target request based on the policy engine integrated in the interceptor corresponding to the target service when the target request arrives at the target service; The log mode is determined based on the log generation policy configured by the policy engine for the target request; The log mode is a predefined structure after configuring the target request based on the log generation strategy; A second processing module, configured to send the log mode of the target request to the intermediate log aggregation system through a policy engine based on the log mode of the target request; A third processing module is used to send log acquisition information to the target service through the intermediate log aggregation system when the intermediate log aggregation system determines that the log needs to be stored; The log acquisition information includes a log template corresponding to the target database; A fourth processing module, configured to generate a target log according to a log template corresponding to the target database through the target service based on the log mode of the target request, and send the target log to the intermediate log aggregation system; The fifth processing module is used to store the target log in the target database through the intermediate log aggregation system.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the program, the log processing method based on the ternary management system as described in any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log processing method based on the ternary management system as claimed in any one of claims 1 to 6 is implemented.

Images