A method of log collection and visualization of a workflow

Abstract

The application discloses a kind of workflow log collection and visualization method, comprising the following steps: S1: receiving log data from all microservices;S2: using AI retrieval, using natural language processing and machine learning for log pattern recognition and correlation analysis;S3: using link tracking framework, keep the context information of request in the whole microservice architecture;S4: using real-time analysis tool, real-time monitoring and anomaly detection are carried out to log stream;S5: in combination with event-driven architecture, automatically trigger alarm notification to corresponding team members for log or mode considered important by analysis;S6: audit current log level and content, optimize log output, ensure the quality and relevance of log data.The application realizes a centralized log aggregation and standardization platform, not only guarantees the integrity and consistency of log, but also significantly reduces the difficulty of log retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log collection, and in particular to a log collection and visualization method of a workflow. BACKGROUND

[0002] Log collection and viewing is a very important part in the field of information technology, which involves collecting, storing and analyzing log data generated by systems and applications. These log data contains various information of system runtime, such as error logs, event logs, performance indicators, etc., which is crucial for monitoring system status, troubleshooting and security auditing. Traditional log collection and viewing methods usually involve manual checking of log files, but with the growth of system size and the increase of log data volume, this method is no longer applicable. Therefore, modern log collection and viewing schemes use automated methods to collect, store and analyze log data.

[0003] Currently, in the scenario centered on microservices, log collection is becoming increasingly difficult, and with the increase and complexity of business processes, there are several pain points as follows:

[0004] 1. Difficulty in log filtering: Log is stored discretely, making it difficult to quickly and completely search for all logs

[0005] 2. Time-consuming log analysis: When problems occur, only the searched logs can be analyzed in combination with the code, or as much printed logs as possible, which consumes labor

[0006] 3. Incomplete log collection: Asynchronous thread execution may lose link information, or the thread pool needs to be rewritten.

[0007] Therefore, the existing technology has defects and needs to be improved. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a log collection and visualization method of a workflow, which overcomes the multiple limitations of current log collection and shortens log analysis from hours to minutes.

[0009] The technical solution of the present application is as follows: a log collection and visualization method of a workflow is provided, comprising the following steps.

[0010] S1: receiving log data from all microservices; all log data uses a unified log format, and the log data is standardized and serialized to facilitate searching and storage.

[0011] S2: using AI retrieval, using natural language processing (NLP) and machine learning for pattern recognition and correlation analysis of logs, so that users can query logs in a more natural way, and provide relevant log aggregation to reduce the workload of manual searching.

[0012] S3: Adopt link tracking framework to maintain the context information of the request throughout the microservice architecture, ensuring that the coherence of the logs is not lost even when asynchronous threads are executed; optionally, modify the standard thread library or use a custom thread pool to ensure the integrity of log collection.

[0013] S4: Use real-time analysis tools to monitor and detect anomalies in log streams in real time, and use custom dashboards and visualization tools to immediately display key performance indicators and security warnings. Real-time analysis tools such as Grafana, Kibana. Grafana: an open-source time series analysis and visualization tool, particularly suitable for monitoring system metrics and measures. Kibana: an open-source analysis and visualization platform based on Elasticsearch, which can help users freely explore, analyze, and visualize data in Elasticsearch indexes.

[0014] S5: Combine event-driven architecture to automatically trigger alarm notifications to relevant team members for logs or patterns deemed important by analysis, through email, SMS, or integrated notification systems.

[0015] S6: Audit the current log level and content, optimize log output, and ensure the quality and relevance of log data; at the same time, manage log data retention and compliance to ensure the security of sensitive data.

[0016] In step S3, modify the standard thread library or use a custom thread pool to ensure the integrity of log collection.

[0017] The specific implementation of step S1 includes the following steps: S11: Adopt a lightweight log collection agent and deploy it on each microservice node to collect log data generated by each service. Use open-source tools such as Logstash or Fluentd, or self-developed collectors that comply with the unified log format specification. S12: Build a log server cluster: use Elasticsearch or Apache Kafka as a centralized log storage and buffer layer to ensure high availability and horizontal scalability; configure data indexing strategies to optimize storage efficiency. S13: Log standardization and serialization: develop a unified log format standard (such as JSON) and apply it in the collection stage to facilitate subsequent parsing and searching. Use serialization protocols such as Avro or Protobuf to improve data transmission efficiency. Data indexing strategies can include: B+ tree index, hash index, full-text index, combined index, clustered index, non-clustered index, covering index, prefix index, spatial index, index merging, etc.

[0018] The specific implementation of step S2 includes the following steps: S21: integrate natural language processing (NLP) engine: select or develop a natural language processing-based search engine, such as Elasticsearch's advanced query function, to support users querying logs in natural language form. S22: machine learning model training: based on historical log data, train a machine learning model to identify log patterns, anomalies, and correlations; frameworks such as TensorFlow or PyTorch can be used, focusing on training classification and clustering algorithms. S23: intelligent recommendation and associated log display: based on query statements and model predictions, provide users with the most relevant log segments and associated log chains, reducing manual troubleshooting time.

[0019] The specific implementation of step S3 includes the following steps: S31: introduce OpenTracing / OpenTelemetry: integrate mature link tracking frameworks such as Jaeger and Zipkin to ensure that request IDs are transparently passed across service calls and maintain log context consistency. S32: custom thread management: develop or customize thread pool management modules for specific scenarios to ensure the continuity and integrity of log recording in asynchronous operations, avoiding log segmentation or loss.

[0020] The specific implementation of step S4 includes the following steps: S41: real-time stream processing engine: use Apache Flink or Spark Streaming technology to process real-time log streams, enabling anomaly detection and performance metric calculation. S42: custom dashboard development: based on Kibana or Grafana tools, design intuitive dashboard interfaces to display key performance indicators (KPIs), security alerts, and trend analysis.

[0021] The specific implementation of step S5 includes the following steps: S51: in the alarm rule configuration system, allow users to customize alarm rules based on log content, frequency, and severity conditions. S52: integrate notification channels: interface with email services, SMS gateways, and enterprise IM tools (such as Slack and DingTalk) to implement multi-channel alarm notification and ensure rapid response in emergency situations.

[0022] The specific implementation of step S6 includes the following steps: S61: log auditing and optimization: regularly review log output, adjust log levels according to business needs, remove redundant logs, and ensure that log content is concise and valuable. S61: data retention and compliance: implement data retention policies, such as time or space threshold-based automatic archiving or deletion of old logs. Ensure that the log processing process complies with data protection regulations such as GDPR and HIPAA.

[0023] With the above scheme, the application provides a workflow log collection and visualization method, which has the following technical effects: for the log collection difficulty in the microservice architecture, by implementing a centralized log aggregation and standardization platform, not only the integrity and consistency of the log are ensured, but also the difficulty of log retrieval is significantly reduced. By using AI retrieval technology, the application can perform deep log analysis, thereby shortening the problem positioning time from the hour level to the minute level. By integrating advanced link tracking technology, the application can ensure that complete log chains can be collected even in complex asynchronous execution processes. In addition, the application uses real-time log analysis and dynamic visualization tools, as well as an automatic alarm mechanism based on events, which provides strong support for ensuring the robustness and maintainability of the microservice architecture. The application can effectively solve the log management challenge in the microservice architecture and improve the system observability, troubleshooting efficiency and security. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The method flowchart of the application. DETAILED DESCRIPTION

[0025] The application will be described in detail below in combination with the drawings and specific embodiments.

[0026] Please refer to Figure 1 The application provides a workflow log collection and visualization method, which includes the following steps.

[0027] S1: receiving log data from all microservices; all log data uses a unified log format, and the log data is standardized and serialized to facilitate searching and storage.

[0028] S2: using AI retrieval, using natural language processing (NLP) and machine learning for log pattern recognition and correlation analysis, so that users can query logs in a more natural way, and provide relevant log aggregation to reduce the workload of manual searching.

[0029] S3: using a link tracking framework to maintain the context information of the request in the entire microservice architecture, ensuring that the coherence of the log will not be lost even when asynchronous threads are executed; optionally, the standard thread library is modified or a custom thread pool is used to ensure the integrity of log collection.

[0030] S4: Real-time monitoring and anomaly detection of log streams using real-time analysis tools, and immediate presentation of key performance indicators and security alerts using custom dashboards and visualization tools. Real-time analysis tools such as Grafana, Kibana. Grafana: an open-source time series analysis and visualization tool, especially suitable for monitoring system metrics and measures. Kibana: an open-source analysis and visualization platform based on Elasticsearch, which can help users freely explore, analyze and visualize data in Elasticsearch indexes.

[0031] S5: In combination with the event-driven architecture, automatically trigger alarm notifications to the corresponding team members for logs or patterns considered important by analysis, through email, SMS or integrated notification systems.

[0032] S6: Audit the current log level and content, optimize log output, and ensure the quality and relevance of log data; at the same time, manage log data retention and compliance to ensure the security of sensitive data.

[0033] In step S3, the standard thread library is modified or a custom thread pool is used to ensure the integrity of log collection.

[0034] The specific implementation of step S1 includes the following steps: S11: Use lightweight log collection agents deployed on each microservice node to collect log data generated by each service. Use open-source tools such as Logstash or Fluentd, or self-developed collectors that comply with the unified log format specification. S12: Build a log server cluster: use Elasticsearch or Apache Kafka as a centralized log storage and buffer layer to ensure high availability and horizontal scalability; configure data indexing strategies to optimize storage efficiency. S13: Log standardization and serialization: develop a unified log format standard (such as JSON) and apply it in the collection stage to facilitate subsequent parsing and searching. Use serialization protocols such as Avro or Protobuf to improve data transmission efficiency. Data indexing strategies can include: B+ tree index, hash index, full-text index, combined index, clustered index, non-clustered index, covering index, prefix index, spatial index, index merging, etc.

[0035] The specific implementation of step S2 includes the following steps: S21: integrate natural language processing (NLP) engine: select or develop a natural language processing-based search engine, such as Elasticsearch's advanced query function, to support users querying logs in natural language form. S22: machine learning model training: based on historical log data, train a machine learning model to identify log patterns, anomalies, and correlations; frameworks such as TensorFlow or PyTorch can be used, focusing on training classification and clustering algorithms. S23: intelligent recommendation and associated log display: based on query statements and model predictions, provide users with the most relevant log segments and associated log chains, reducing manual troubleshooting time.

[0036] The specific implementation of step S3 includes the following steps: S31: introduce OpenTracing / OpenTelemetry: integrate mature link tracking frameworks such as Jaeger and Zipkin to ensure that request IDs are transparently passed across service calls and maintain log context consistency. S32: custom thread management: develop or customize thread pool management modules for specific scenarios to ensure the continuity and integrity of log recording in asynchronous operations, avoiding log segmentation or loss.

[0037] The specific implementation of step S4 includes the following steps: S41: real-time stream processing engine: use Apache Flink or Spark Streaming technology to process real-time log streams, enabling anomaly detection and performance metric calculation. S42: custom dashboard development: based on Kibana or Grafana tools, design intuitive dashboard interfaces to display key performance indicators (KPIs), security alerts, and trend analysis.

[0038] The specific implementation of step S5 includes the following steps: S51: in the alarm rule configuration system, allow users to customize alarm rules based on log content, frequency, and severity conditions. S52: integrate notification channels: interface with email services, SMS gateways, and enterprise IM tools (such as Slack and DingTalk) to implement multi-channel alarm notification and ensure rapid response in emergency situations.

[0039] The specific implementation of step S6 includes the following steps: S61: log auditing and optimization: regularly review log output, adjust log levels according to business needs, remove redundant logs, and ensure that log content is concise and valuable. S61: data retention and compliance: implement data retention policies, such as time or space threshold-based automatic archiving or deletion of old logs. Ensure that the log processing process complies with data protection regulations such as GDPR and HIPAA

[0040] In summary, the present application provides a workflow log collection and visualization method, which has the following technical effects: for the log collection difficulty in microservice architecture, by implementing a centralized log aggregation and standardization platform, not only the integrity and consistency of the log are guaranteed, but also the difficulty of log retrieval is significantly reduced. By using AI retrieval technology, the present application can perform deep log analysis, thereby shortening the problem positioning time from hours to minutes. By integrating advanced link tracking technology, the present application can ensure that even in a complex asynchronous execution process, complete log chains can be collected. In addition, the present application uses real-time log analysis and dynamic visualization tools, as well as an event-based automatic alarm mechanism, which provides strong support for ensuring the robustness and maintainability of the microservice architecture. The present application can effectively solve the log management challenge in the microservice architecture, and improve the system observability, troubleshooting efficiency and security.

[0041] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for log collection and visualization of a workflow, characterized in that, The method comprises the following steps: S1: receiving log data from all microservices; all log data uses a unified log format, and the log data is standardized and serialized to facilitate searching and storage; S2: using AI retrieval, using natural language processing and machine learning for pattern recognition and correlation analysis of logs; S3: using a link tracking framework to maintain the context information of a request in the entire microservice architecture, ensuring that the coherence of the logs is not lost even when asynchronous threads are executed; S4: using real-time analysis tools to monitor and detect anomalies in log streams in real time, and using custom dashboards and visualization tools to immediately display key performance indicators and security warnings; S5: combining an event-driven architecture, automatically triggering alarm notifications to the corresponding team members for logs or patterns considered important by analysis, through email, SMS, or integrated notification systems; S6: auditing the current log level and content, optimizing log output; at the same time, data retention and compliance management are performed on the log data; The specific implementation of step S2 comprises the following steps: S21: integrate a natural language processing engine: select or develop a search engine based on natural language processing to support users to query logs in natural language form; S22: machine learning model training: based on historical log data, train a machine learning model to identify log patterns, anomalies, and correlations; use TensorFlow or PyTorch framework to train classification and clustering algorithms; S23: intelligent recommendation and associated log display: according to the query statement and model prediction, provide the most relevant log fragments and associated log chains for the user.

2. The workflow log collection and visualization method of claim 1, wherein, In step S3, the standard thread library is modified or a custom thread pool is used to ensure the integrity of log collection.

3. The method of claim 1, wherein, The specific implementation of step S1 comprises the following steps: S11: use a lightweight log collection agent, deployed on each microservice node, to collect log data generated by each service; S12: build a log server cluster: use Elasticsearch or Apache Kafka as a centralized log storage and buffer layer; and configure data indexing strategies to optimize storage efficiency; S13: log standardization and serialization: develop a unified log format standard and apply it in the collection stage to facilitate subsequent parsing and searching; use serialization protocols to improve data transmission efficiency.

4. The method of claim 1, wherein, The specific implementation of step S3 comprises the following steps: S31: introduce OpenTracing / OpenTelemetry: integrate a mature link tracking framework to ensure that the request ID is transparently passed across service calls, maintaining log context consistency; S32: custom thread management: develop or customize thread pool management modules for specific scenarios to ensure the continuity and completeness of log recording in asynchronous operations.

5. The method of claim 1, wherein, The specific implementation of step S4 comprises the following steps: S41: real-time stream processing engine: use Apache Flink or Spark Streaming technology to process real-time log streams, implement anomaly detection and performance indicator calculation S42: Custom Dashboard Development: Utilize tools like Kibana or Grafana to design intuitive dashboard interfaces that display key performance indicators, security alerts, and trend analysis.

6. The workflow log collection and visualization method of claim 1, wherein, The specific implementation of step S5 includes the following steps: S51: In the alarm rule configuration system, allow users to customize alarm rules based on log content, frequency, severity conditions; S52: Integration of notification channels: interface with email services, SMS gateways, and enterprise IM tools to achieve multi-channel alarm notification and ensure rapid response in emergency situations.

7. The workflow log collection and visualization method of claim 1, wherein, The specific implementation of step S6 includes the following steps: S61: Log auditing and optimization: Regularly review log output, adjust log levels according to business needs, and remove redundant logs; S62: Data retention and compliance: Implement data retention policies.

8. The workflow log collection and visualization method of claim 7, wherein, In step S61, the data retention policy is to automatically archive or delete old logs based on time or space thresholds.

Images