Systems and methods for log tracing and monitoring specific data in distributed service systems

Abstract

A method for log tracking and monitoring specific data in a distributed service system includes: providing a server to receive multiple logs of different formats from multiple sources; providing a first module communicatively connected to the server and configured to define a log format and identify at least one identification data among the multiple logs; the first module programming each of the multiple logs by placing the at least one identification data into the log format to generate multiple first data; and providing a second module communicatively connected to the first module to receive the multiple first data, such that the second module identifies information of the multiple first data based on the identification data.

Description

Technical Field

[0001] This invention relates to a system and method for log tracking and monitoring specific data in a distributed service system, and more particularly to a system and method for tracking and monitoring specific logs in a distributed service system. Prior Technology

[0002] When multiple users operate the system simultaneously and multiple hosts run at the same time, and the system architecture is divided into multiple sub-modules or microservices, the inability to centrally monitor specific logs will result in some significant drawbacks.

[0003] First, log information is scattered across different modules and hosts, making it impossible to effectively integrate the data. To track the behavior of a single user, it may be necessary to extract data from multiple sources and perform tedious comparisons, which not only increases the difficulty of analysis but also wastes a lot of time and resources.

[0004] Secondly, when logs are scattered across multiple modules and cannot be centrally managed, it becomes difficult to track the behavior of specific users or operators in real time, affecting the system's response speed when detecting problems and anomalies. This is particularly inconvenient when dealing with emergencies or troubleshooting, because if the operation trajectory of a specific user cannot be quickly found, the system administrator will find it difficult to determine the root cause of the problem, thus delaying system repair. For example, if a user performs an improper operation, such as injecting incorrect code, the scattered logs may prevent the administrator from quickly finding the user's complete operation record, prolonging the investigation process and thus affecting system stability.

[0005] Furthermore, the lack of centralized log monitoring capabilities also poses security risks. Without the ability to track and analyze the behavior of specific users, system administrators may be unable to promptly detect abnormal behavior when security incidents occur (such as unauthorized logins or malicious operations), increasing the risk of attackers or malicious users maintaining a long-term presence in the system. This renders intrusion detection and security incident response inefficient, reduces system security, and may even lead to data breaches or compliance issues. In addition, if administrators cannot quickly locate the abnormal behavior of specific users, it may affect the overall security level of the system and pose a potential threat to other users.

[0006] Finally, when logs cannot be centrally managed and analyzed for specific targets (such as individual users or operators), management struggles to monitor system operations for compliance and cannot ensure that operational processes conform to regulations. This not only hinders post-incident traceability but may also increase the risk of internal operational violations, especially in industries requiring stringent compliance (such as finance or healthcare). Centralized log monitoring allows managers to quickly detect abnormal behavior, ensure compliance, and improve system management efficiency, while the inability to effectively track individual users or operators weakens management and security controls.

[0007] To address the aforementioned issues, the conventional approach is to manually add specific identification markers to each log entry, enabling managers to track and monitor the logs based on these markers. Summary of the Invention

[0008] To overcome the aforementioned shortcomings, this invention provides a simpler solution. Specifically, this invention uses a system to identify specific identifiers in logs, automatically embedding these identifiers into a specific format, and further reprogramming logs of different formats using this specific log format, enabling administrators to track and monitor specific logs based on these identifiers. The method of this invention can be implemented quickly and is suitable for tracking user operations within a system.

[0009] Based on the above concept, the present invention provides a method for log tracking and monitoring specific data in a distributed service system, comprising: providing a system for receiving multiple logs of different formats from multiple sources; providing a first module, the first module being communicatively connected to the server and configured to define a log format and identify at least one identification data among the multiple logs; the first module placing the at least one identification data into the log format to program each of the multiple logs to generate multiple first data; and providing a second module, the second module being communicatively connected to the first module to receive the multiple first data, such that the second module identifies information of the multiple first data based on the identification data.

[0010] Based on the above concept, the present invention provides a system for log tracking and monitoring specific data in a distributed service system. The system receives multiple logs of different formats from multiple sources. The system includes: a first module communicatively connected to the server and configured to define a log format and identify at least one identification data among the multiple logs; wherein the first module further places the at least one identification data into the log format to program each of the multiple logs, thereby generating multiple first data; and a second module communicatively connected to the first module to receive the multiple first data, such that the second module identifies information from the multiple first data based on the identification data.

[0011] In one embodiment, the log format is log.info("SubscrId = "+scbscrId+" log content") or log.info("UserId="+userId+", SubscrId="+scbscrId+" log content").

[0012] In one embodiment, the identification data is placed in the front end or back end of the log format.

[0013] In one embodiment, the identification data includes a first identification data associated with a user's identification code.

[0014] In one embodiment, the identification data further includes a second identification data associated with an identification code of a system administrator or programmer. Simple Explanation of the Diagram

[0015] Figure 1 shows a schematic diagram of the system of the present invention.

[0016] Figure 2 shows a schematic diagram of the method of the present invention.

[0017] Figure 3 shows a schematic diagram of the log format of the present invention.

[0018] Figure 4A shows a schematic diagram of the original log of the present invention.

[0019] Figure 4B shows a schematic diagram of the first material of the present invention.

[0020] Figure 5 shows a schematic diagram of the interface of the second module of the present invention. Implementation

[0021] Referring to Figure 1, the system 100 of this invention includes a first module 110 and a second module 120. The system 100 is a server configured to receive multiple logs (log1 to logx) of different formats from multiple sources, where the sources refer to other worker servers or other modules, and the logs (log1 to logx) are generated in different formats based on different work natures or different software used, for example, a worker server. The system 100 of this invention acts as a centrally managed server and communicates with other worker servers or other modules in a distributed system to receive or collect multiple logs of different formats provided by other worker servers or other modules. In this invention, the first module 110 is communicatively connected to the system 100 to reprogram the logs of different formats to generate multiple first data sets, and the second module 120 is communicatively connected to the first module 110 to receive and manage the data of the multiple first data sets.

[0022] In detail, the first module 110 utilizes the thread-local map provided by Logback / Log4j to help manage log context information. More specifically, the first module 110 of this invention employs the MDC (Mapped Diagnostic Context) mechanism, which is essentially a thread-local map used to store key-value information, ensuring that each log contains specific identification data, such as request ID, user ID, etc., facilitating subsequent analysis and querying.

[0023] The second module 120 series is configured to process and manage large-scale log data. In a preferred embodiment of the invention, the second module 120 series may include one or a combination of applications such as Logstash, Elasticsearch, or Kibana to process and manage large-scale log data. Details of applications such as Logstash, Elasticsearch, and Kibana are described below.

[0024] Logstash is an open-source data collection and processing tool used to collect, parse, and transform data from various sources, such as server logs, application logs, and databases. Logstash can process data in different formats, filtering, formatting, and standardizing it before sending the processed data to backend storage (such as Elasticsearch). Logstash supports various input and output plugins, enabling it to adapt to diverse data sources and output destinations, and flexibly combine application rules for data transformation and enhancement. For example, Logstash can standardize different log sources uniformly for better subsequent search and analysis.

[0025] Elasticsearch is a decentralized full-text search and analytics engine specifically designed for fast searching, storage, and querying of large-scale data. Logstash feeds collected and processed data to Elasticsearch, allowing Elasticsearch to store and index this data. Elasticsearch supports a powerful query language and data aggregation capabilities, enabling users to instantly search, filter, and analyze large-scale log data. Furthermore, due to its decentralized architecture, Elasticsearch can scale and process massive amounts of data while ensuring high query efficiency, making it an ideal choice for handling massive log data.

[0026] Kibana is a visualization tool used to interact with Elasticsearch data and create charts and dashboards (see Figure 5). With Kibana, users can visually display and analyze data stored in Elasticsearch. It supports various chart types (such as line charts, pie charts, heatmaps, etc.) to display data trends and statistical results. Furthermore, Kibana's real-time search and filtering capabilities allow users to delve deeper into data, monitor system status, and identify anomalies. This enables system administrators and operations personnel to gain a more intuitive understanding of the system's health and operational behavior through visualization.

[0027] Next, the method for log tracking and monitoring specific data in a distributed service system according to the present invention will be described in detail. Referring to Figure 2, in step S201, the present invention provides a system 100 to receive logs of different formats, log1 to logx, from the aforementioned other worker servers or other modules in a distributed system.

[0028] In step S202, the first module 110 is communicatively connected to the system and configured to define a log format and identify at least one identification data among the plurality of logs. Specifically, the first module 110 defines a log format using the MDC mechanism. Referring to Figure 3, the log format includes at least identification data and log content. The identification data includes first identification data and second identification data. The first identification data is associated with a user's identification code, such as UserId, and the second identification data is associated with a system administrator's identification code, such as SubscrId. The log content is the content of the original log (Message). In a specific embodiment of the present invention, during system operation, UserId and SubscrId serve as core identification information, ensuring the traceability of requests and the relevance of logs. To maintain this information throughout the request processing chain, the system extracts these two identification values ​​from the request context (such as HTTP Headers, Session, or Token) upon receiving a request and stores them in the MDC. The MDC is a mechanism based on ThreadLocal, allowing request-related context information to be bound to the current thread, ensuring that this information is automatically referenced in subsequent log entries without affecting the execution of other requests. In a preferred embodiment of the invention, the log format is at least log.info("SubscrId="+scbscrId+" log content") or log.info("UserId="+userId+", SubscrId="+scbscrId+" log content"). In other embodiments of the invention, the log format may include the timestamp of the original log.

[0029] In step S203, the first module places at least one identification data into the log format to program each of the plurality of logs, thereby generating a plurality of first data. Please refer to Figure 4A for a schematic diagram of the original logs and Figure 4B for a schematic diagram of the first data. Specifically, after placing the identification data (e.g., SubscrId) into the log format defined by the MDC, the first module 110 further reprograms each original log based on the log format to generate the first data. In a specific embodiment of the present invention, after the MDC stores UserId and SubscrId, all logs in the application can obtain this information through the MDC. Log frameworks (such as Logback or Log4j) can automatically insert the content in the MDC into the log format through preset pattern variables (such as %X{UserId} and %X{SubscrId} identification data), ensuring that all recorded logs are labeled with the corresponding user identification code and system administrator identification code. As shown in Figure 4B, using the above methods, the first module 110 can generate first data by attaching identification data such as UserId and SubscrId to the original log. In embodiments of the present invention, the identification data can be placed or set at the front end of the log format. For example, as shown in Figure 4B, the identification data such as UserId and SubscrId are placed at the front end of the log format. In other embodiments of the present invention, the identification data such as UserId and SubscrId can be placed at the back end of the log format, but this is not a limitation. In the present invention, the first data may also include information such as timestamp, log level, log source, request / transaction ID, error stacking, and user / device information.

[0030] Next, in step S204, the second module 120 is communicatively connected to the first module 110 to receive the plurality of first data from the first module 110, so that a user can use the second module and identify information in the plurality of first data based on the identification data. Specifically, the second module 120 processes and manages large-scale first data using applications such as Logstash, Elasticsearch, and Kibana mentioned above. Please refer to Figure 5, which shows a schematic diagram of the interface of the second module 120. Figure 5 specifically shows the interface displayed by the Kibana application. Users or administrators can use Kibana to filter specific information from multiple sets of first data. In a specific embodiment of the present invention, users or administrators can input identification data such as UserId and SubscrId (inputting "SubscrId:3249857" in the upper left corner of the figure). The interface of the second module 120 will then appear as shown in Figure 5. Users or administrators can filter out multiple logs related to "SubscrId:3249857" and use charts to quickly identify which time period has a large number of DEBUGs or modifications made by other users, thereby determining the exact problem point for handling. For example, the figure shows that when the timeline is around 10:50:00, the system of the present invention receives a large number of logs. Users or administrators can select the categories "eventDatetime" and "message" to directly view the logs in "eventDatetime". And in the "message" field (shown by the dotted line), you can quickly obtain information from the logs to identify those that have problems or need to be modified, so as to quickly manage a large amount of primary data.

[0031] Overall, the method and system for log tracking and monitoring specific data in distributed service systems provided by this invention utilize the MDC mechanism to define a log format, thereby attaching identification data such as UserId and SubscrId to the original logs. This solves the problem of manually adding specific identification markers to each different format log using conventional methods. Furthermore, according to the mechanism of this invention, system administrators can quickly locate operations related to specific users through logs, improving the efficiency of troubleshooting and data auditing.

[0032] 100: System

[0033] 110: First Module

[0034] 120: Second Module

[0035] log1-logx: logs

[0036] S201-S204: Steps

Claims

1. A method for log tracking and monitoring specific data in a distributed service system, comprising: providing a system for receiving multiple logs of different formats from multiple sources; providing a first module communicatively connected to the system and configured to define a log format and identify at least one identification data among the multiple logs; the first module placing the at least one identification data into the log format to program each of the multiple logs to generate multiple first data; providing a second module communicatively connected to the first module to receive the multiple first data, such that the second module identifies information of the multiple first data based on the identification data; wherein the log format is in the form of log.info("SubscrId="+scbscrId+" log content") or log.info("UserId="+userId+", SubscrId="+scbscrId+" log content"); wherein the identification data is placed in the front-end or back-end of the log format; The identification data includes a first identification data associated with a user's identification code; the identification data also includes a second identification data associated with a system administrator's identification code.

2. A system for log tracking and monitoring specific data in a distributed service system, the system receiving multiple logs of different formats from multiple sources, the system comprising: a first module configured to define a log format and identify at least one identification data among the multiple logs, wherein the first module further places the at least one identification data into the log format to program each of the multiple logs to generate multiple first data; a second module communicatively connected to the first module to receive the multiple first data, such that the second module identifies information of the multiple first data based on the identification data; wherein the log format is in the form of log.info("SubscrId="+scbscrId+" log content") or log.info("UserId="+userId+", SubscrId="+scbscrId+" log content"); wherein the identification data is placed at the front end or back end of the log format; wherein the identification data has a first identification data associated with a user's identification code. The identification data also includes a second identification data, which is associated with an identification code of a system administrator or programmer.

Images