A segmented log persistence system and method for enterprise service bus

Abstract

The present invention discloses a segmented log persistence system and method for an enterprise service bus, belonging to the field of information technology technology. The system comprises a log generation module, a built-in queue, a distributed message queue, a log combination module, and a log storage module. The system processes resource logs in multiple segments so that when an exception or response timeout occurs at a capability node, the log storage module can obtain the completed log information of normal capability nodes, and store it in Elasticsearch after processing, so that operation and maintenance personnel can track and troubleshoot exceptions in a timely manner and quickly locate problems. The log persistence is also performed asynchronously via a built-in queue in a disruptor. That is, when the Kafka message queue consumes log information, the system's gateway call and log processing are performed synchronously, greatly improving the efficiency of service gateway calls and log storage.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a segmented log persistence system and method for an enterprise service bus. Background Art

[0002] With the rapid development of microservices, more and more services are being deployed, the calls between services are becoming more and more complex, and the connections between them are becoming increasingly close. When a service has a problem, it is becoming increasingly difficult to troubleshoot the problem, causing operations and developers to spend a lot of time searching and locating logs. To this end, a segmented log persistence system and method for an enterprise service bus is proposed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is: when an exception occurs in a capability node or the response times out, how to obtain the log information of the completed normal capability node, so that the operation and maintenance personnel can track and troubleshoot the exception in time and quickly locate the problem. A log segmentation persistence system for enterprise service bus is provided.

[0004] The present invention solves the above technical problems through the following technical solutions, which include a log generation module, a built-in queue, a distributed message queue, a log combination module, and a log storage module;

[0005] The log generation module is used to generate a request reception log, a capability node processing log, and a request response log at the beginning, processing, and end of each request, and store the request reception log, capability node processing log, and request response log in three different built-in queue objects respectively;

[0006] The built-in queue is used to configure producer and consumer modes to send, receive, and process log messages;

[0007] The distributed message queue is used to take out the log objects at the start, processing and end of the request from the built-in queue for consumption;

[0008] The log combination module is used to consume log information from Kafka and update or add logs in the log storage module according to the log ID, completing the combination of multi-segment logs for log objects corresponding to the same log ID;

[0009] The log storage module is used to collect and store log objects generated according to log IDs in the log combination module for platform operation and maintenance personnel to retrieve and read the logs.

[0010] Furthermore, in the log generation module, each log generated by a request has a same log ID.

[0011] Furthermore, in the log generation module, at the beginning of the request, a request reception log is generated based on the service name, service code, application name and application code associated with the service, authentication method, and persistence policy information obtained from the service gateway; during capability node processing, a capability node processing log is generated based on the node type, node name, message list, message body, and message size information of each capability node included in the request service; at the end of the request, a request response log is generated based on the time of processing this service, the time of receiving the message, and the time information of the log generation.

[0012] Furthermore, the built-in queue is a built-in message queue framework of the disruptor, which stores the request reception log, capability node processing log, and request response log in the log generation module into the built-in RingBuffer ring queue in its framework to realize asynchronous processing of log information.

[0013] Furthermore, the distributed message queue is a Kafka message queue, and each built-in queue object corresponds to a different topic in the Kafka message queue.

[0014] Furthermore, in the log combination module, log information is consumed from the Kafka message queue, and a complete log information including request reception log, capability node processing log, and request response log is generated according to the log ID.

[0015] Furthermore, the log storage module is implemented based on the distributed full-text search engine elasticsearch, and has the ability to search in real time, analyze and process logs.

[0016] The present invention also provides a segmented log persistence method for an enterprise service bus, which uses the above system to implement segmented log persistence, including the following steps:

[0017] S1: Logs are generated at the beginning of each request, when the capability node processes the request, and when the request ends. The logs generated by each request have the same log ID. The request reception log, capability node processing log, and request response log are stored in three different built-in queue objects respectively.

[0018] S2: S2: Uses the disruptor's built-in message queue to receive and process message logs;

[0019] S3: Consumes log objects in the built-in queue through the Kafka message queue;

[0020] S4: Consume log information from the Kafka message queue and update or add logs in the log storage module according to the log ID, completing the combination of multi-segment logs for log objects corresponding to the same log ID;

[0021] S5: Collect the log objects generated according to the log ID in the log combination module and store them in the log storage module.

[0022] Furthermore, step S2 includes the following sub-steps:

[0023] S21: Create a bean of the RingBuffer class in the disruptor framework in Spring, which is the built-in message queue object;

[0024] S22: Create a built-in message queue producer DisruptorProducer object, call the publishEvent method of the RingBuffer object in the DisruptorProducer object, and use the log data as the parameter to send the data to the built-in message queue;

[0025] S23: Create a built-in message queue producer object, call the publishEvent method of the RingBuffer object in the producer object, and use the log data as the parameter to send the data to the built-in message queue;

[0026] S24: Create a built-in message queue consumer object. The consumer inherits the EventHandler class in the disruptor framework and rewrites the onEvent method to consume the log data in the built-in message queue.

[0027] Compared with the existing technology, the present invention has the following advantages: the segmented log persistence system and method for the enterprise service bus performs multi-segment processing on resource logs so that when an exception occurs or a response times out at a certain capability node, the log storage module can obtain the completed normal capability node log information, and store it in elasticsearch after processing, so that operation and maintenance personnel can track and investigate the exception in time and quickly locate the problem; and the log persistence method is asynchronous through the built-in queue of the disruptor, that is, when the Kafka message queue consumes the log information, the system calls the gateway and processes the log synchronously, which greatly improves the efficiency of service gateway calls and log storage, and is worthy of promotion and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural diagram of the log segmentation persistence system in Example 1 of the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0030] Example 1

[0031] like Figure 1 As shown, this embodiment provides a technical solution: a segmented log persistence system for an enterprise service bus, including a log generation module, a built-in queue, a Kafka message queue, a log combination module, and an elasticsearch log storage module.

[0032] In this embodiment, in the log generation module, at the beginning of a request, the service name, service code, service-associated application name and application code, authentication method, persistence strategy, and other field information obtained from the service gateway are initialized to form a request reception log. When processing a capability node, the node type, node name, message list, message body, message size, and other field information of each capability node included in the request service (especially: if the node type is an HTTP resource, the request header, request URL, and other field information are included; if the node type is a Groovy script, the script processing time and other field information are included) are used to form a capability node processing log. At the end of the request, the time of processing the service, the time of receiving the message, the time of log generation, and other field information are filled in to form a request response log.

[0033] In this embodiment, the disruptor's built-in message queue framework is a high-performance queue for multiple threads within a JVM (Java Virtual Machine), enabling concurrent network queue operations without locking. A producer mode is defined through a configuration class to send messages, and a consumer processor is configured to process messages sent by the producer. This module stores the request reception log, capability node processing log, and request response log from the log generation module in the framework's built-in RingBuffer queue, achieving asynchronous log processing.

[0034] In this embodiment, in the Kafka message queue (distributed message queue), the Kafka message queue is used to take out the log objects at the start, processing, and end of the request from the built-in queue for consumption. Each built-in queue object corresponds to a different topic in the Kafka message queue.

[0035] In this embodiment, in the log combination module, this module consumes log information from Kafka and generates a complete log information including request reception log, capability node processing log, and request response log according to the log ID.

[0036] In this embodiment, the elasticsearch log storage module is a distributed full-text search engine that provides real-time search, reliable, stable, and fast analysis and processing capabilities for massive logs.

[0037] This embodiment also provides a segmented log persistence method based on the enterprise service bus, including the following steps:

[0038] S1: The services of the enterprise service bus are composed of multiple capabilities. Logs are generated at the beginning, processing, and end of each request. Each log generated by a request has the same log ID. The request reception log, capability node processing log, and request response log are stored in three different built-in queue objects respectively.

[0039] S2: Use the built-in message queue framework to configure producer and consumer modes to send, receive, and process message logs;

[0040] S3: Consumes log objects in the built-in queue through the Kafka message queue;

[0041] S4: Consume log information from the Kafka message queue and update or add logs in the elasticsearch log storage module according to the log ID, completing the combination of multi-segment logs for log objects corresponding to the same log ID;

[0042] S5: Collect the log objects generated by the log ID in the log combination module and store them in the elasticsearch log storage module;

[0043] In this embodiment, step S2 includes the following sub-steps:

[0044] S21: Create a bean of the RingBuffer class in the disruptor framework in Spring, which is the built-in message queue object;

[0045] S22: Create a built-in message queue producer DisruptorProducer object, call the publishEvent method of the RingBuffer object in the DisruptorProducer object, and use the log data as the parameter to send the data to the built-in message queue;

[0046] S23: Create a built-in message queue producer object, call the publishEvent method of the RingBuffer object in the producer object, and use the log data as the parameter to send the data to the built-in message queue;

[0047] S24: Create a built-in message queue consumer object. The consumer inherits the EventHandler class in the disruptor framework and rewrites the onEvent method to consume the log data in the message queue.

[0048] Example 2

[0049] In this embodiment, the user requests service A, the service code is serviceA, and the caller is consumerA. After the request is completed, the call log is queried. The call log is divided into three parts: when the request starts, when the capability node processes, and when the request ends. Each part records information such as the request header, request message content, request message size, and request duration. It also records information such as the caller name, service name, service code, and call time, which intuitively and clearly reflects the entire call process.

[0050] Except for the above implementation, the rest of the implementation in this embodiment is the same as that in the first embodiment.

[0051] To sum up, the segmented log persistence system for the enterprise service bus in the above embodiment performs multi-segment processing on the resource log so that when an exception occurs or the response times out in a certain capability node, the log storage module can obtain the completed normal capability node log information, and store it in elasticsearch after processing, so that operation and maintenance personnel can track and investigate the exception in time and quickly locate the problem; and the log persistence method is asynchronous through the built-in queue of the disruptor, that is, when the kafka message queue consumes the log information, the system calls the gateway and processes the log synchronously, which greatly improves the efficiency of service gateway calls and log storage, and is worthy of promotion and use.

[0052] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A segmented log persistence system for an enterprise service bus, characterized in that: include: Log generation module, built-in queue, distributed message queue, log combination module, log storage module; The log generation module is used to generate a request reception log, a capability node processing log, and a request response log at the beginning, processing, and end of each request, and store the request reception log, capability node processing log, and request response log in three different built-in queue objects respectively; The built-in queue is used to configure producer and consumer modes to send, receive, and process log messages; The distributed message queue is used to take out the log objects at the start, processing and end of the request from the built-in queue for consumption; The log combination module is used to consume log information from Kafka and update or add logs in the log storage module according to the log ID, completing the combination of multi-segment logs for log objects corresponding to the same log ID; The log storage module is used to collect and store log objects generated according to log IDs in the log combination module for platform operation and maintenance personnel to retrieve and read the logs.

2. The segmented log persistence system for an enterprise service bus according to claim 1, characterized in that: In the log generation module, each log generated by a request has a same log ID.

3. The segmented log persistence system for an enterprise service bus according to claim 2, characterized in that: In the log generation module, at the beginning of a request, a request reception log is generated based on the service name, service code, application name and application code associated with the service, authentication method, and persistence policy information obtained from the service gateway; during capability node processing, a capability node processing log is generated based on the node type, node name, message list, message body, and message size information of each capability node included in the requested service; at the end of the request, a request response log is generated based on the time of processing this service, the time of receiving the message, and the time information of the log generation.

4. The segmented log persistence system for an enterprise service bus according to claim 3, characterized in that: The built-in queue is the disruptor's built-in message queue framework, which stores the request reception log, capability node processing log, and request response log in the log generation module into the RingBuffer ring queue built into its framework to realize asynchronous processing of log information.

5. The segmented log persistence system for an enterprise service bus according to claim 4, characterized in that: The distributed message queue is a Kafka message queue, and each built-in queue object corresponds to a different topic in the Kafka message queue.

6. The segmented log persistence system for an enterprise service bus according to claim 5, characterized in that: In the log combination module, log information is consumed from the Kafka message queue, and a complete log information including request reception log, capability node processing log, and request response log is generated according to the log ID.

7. The segmented log persistence system for an enterprise service bus according to claim 6, characterized in that: The log storage module is implemented based on the distributed full-text search engine elasticsearch, and has the ability to search in real time, analyze and process logs.

8. A method for segmented log persistence for an enterprise service bus, wherein the method adopts the system according to any one of claims 1 to 7 to implement segmented log persistence, comprising the following steps: S1: Logs are generated at the beginning of each request, when the capability node processes the request, and when the request ends. The logs generated by each request have the same log ID. The request reception log, capability node processing log, and request response log are stored in three different built-in queue objects respectively. S2: Use the disruptor's built-in message queue to receive and process message logs; S3: Consumes log objects in the built-in queue through the Kafka message queue; S4: Consume log information from the Kafka message queue and update or add logs in the log storage module according to the log ID, completing the combination of multi-segment logs for log objects corresponding to the same log ID; S5: Collect the log objects generated according to the log ID in the log combination module and store them in the log storage module.

9. The segmented log persistence method for an enterprise service bus according to claim 8, characterized in that: The step S2 includes the following sub-steps: S21: Create a bean of the RingBuffer class in the disruptor framework in Spring, which is the built-in message queue object; S22: Create a built-in message queue producer DisruptorProducer object, call the publishEvent method of the RingBuffer object in the DisruptorProducer object, and use the log data as the parameter to send the data to the built-in message queue; S23: Create a built-in message queue producer object, call the publishEvent method of the RingBuffer object in the producer object, and use the log data as the parameter to send the data to the built-in message queue; S24: Create a built-in message queue consumer object. The consumer inherits the EventHandler class in the disruptor framework and rewrites the onEvent method to consume the log data in the built-in message queue.

Images