Ebpf-based message middleware message delivery diagnosis method and system

Abstract

The application discloses an eBPF-based message middleware message delivery diagnosis method and system, and belongs to the technical field of computers.The method comprises the following steps: deploying an eBPF probe in the operating system kernel of a message middleware client and a server, and mounting the eBPF probe at the entry point and return point of message production, consumption, putting and taking functions; capturing four-level events of message production, putting, taking and consumption, extracting a message unique identifier and a queue name, and generating a timestamp; delivering event data to a user state through an eBPF mapping with the message unique identifier as a key; maintaining a four-level event cache and a message state table in the user state, and performing event correlation matching with the message unique identifier as an index; when a consumption event is received, determining that delivery is successful and calculating a delay, and when a consumption event is not received within a timeout, determining that delivery is abnormal.The application realizes zero-intrusion tracking of the whole life cycle of a single message, can accurately locate a delivery fault link, and can calculate end-to-end segment delay.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a message delivery diagnosis method and system based on eBPF message middleware. Background Technology

[0002] TongLink / Q, a widely used domestic messaging middleware in the financial information technology innovation field, holds a dominant position in core business systems such as payment clearing and cross-border settlement. However, its native monitoring system mainly provides aggregated status monitoring at the system and queue levels, such as node running status, queue depth, and total message count, making it difficult to achieve accurate end-to-end lifecycle tracking of a single message from production and storage to consumption.

[0003] In the prior art, CN116069608A discloses an application call chain tracing method based on eBPF. This method generates a unique link identifier by triggering an eBPF program in the kernel, associates the basic information and response information of the request source application and the target application, and writes this information to a message queue to achieve link tracing. This solution initially achieves zero-intrusion call chain collection based on eBPF, but its technical solution still has the following shortcomings: First, this solution can only capture event information of application layer request calls and responses, and cannot delve into the message middleware to obtain event data of key flow links such as message enqueue (Put) and dequeue (Get) on the server side. As a result, the message processing process inside the middleware is still in a "black box" state, making it difficult to accurately locate whether the fault occurs on the production end, the server end, or the consumer end.

[0004] Second, the scheme does not disclose the technical means of multi-stage event association matching in user space based on the unique message identifier (MsgId), and cannot calculate the precise end-to-end delay of a single message from production to consumption, nor can it achieve real-time proactive diagnosis and alarm for message delivery timeout or loss.

[0005] Therefore, how to accurately capture, cross-platform correlation matching, and real-time diagnosis of delivery status of a single message's entire lifecycle without modifying the message middleware and business application code has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the technical problems existing in the background art described above, the present invention provides a message delivery diagnosis method and system based on eBPF message middleware.

[0007] This invention provides a message delivery diagnosis method based on eBPF message middleware, comprising the following steps: Step S1: Deploy an eBPF probe program in the operating system kernel running the message middleware client and server. The eBPF probe program is attached to the entry and return points of the message production and message consumption functions in the message middleware client library through the uprobe mechanism, and is also attached to the entry and return points of the message insertion and message retrieval functions in the message middleware server program through the uprobe mechanism.

[0008] Step S2: Capture message production events, message placement events, message retrieval events, and message consumption events through the eBPF probe program, extract the unique message identifier and queue or topic name from the function parameter structure corresponding to each event, and generate the timestamp corresponding to each event.

[0009] Step S3: Write the captured event data into the eBPF mapping in key-value pairs. The event data is then transferred from kernel space to user space through the eBPF mapping. The key is the unique identifier of the message, and the value includes the event type, queue or topic name, and timestamp information.

[0010] Step S4: In user space, continuously read event data from the eBPF mapping through the interface provided by eBPF, and send the read event data to the correlation analysis processing flow.

[0011] Step S5: Maintain message production event cache, message placement event cache, message retrieval event cache, and message consumption event cache in user space, as well as a message status table; store each received event record into the corresponding cache according to its event type, and search the message status table for a record corresponding to the unique message identifier; if it does not exist, create a new record in the message status table and store the current event type and timestamp; if it exists, update the corresponding event type and timestamp in the message status table; when a consumption event is received, calculate the end-to-end delivery delay and segmentation delay based on the production time, placement time, retrieval time, and consumption time recorded in the message status table, and determine that the message delivery was successful; if no consumption event is recorded in the message status table within the preset timeout window, determine that the message delivery is abnormal.

[0012] Step S6: Push the diagnostic results and delay indicators to the unified monitoring system for visualization and alarm. The diagnostic results include successful delivery records, delivery anomaly alarms, and delivery node anomaly alarms.

[0013] This invention also provides a message delivery diagnostic system based on eBPF message middleware for implementing the above method, the system comprising: The eBPF probe module is deployed in the operating system kernel running the message middleware client and server. It is used to attach to the entry and return points of message production function, message consumption function, message insertion function and message retrieval function through the uprobe and uretprobe mechanisms, capture message production event, message insertion event, message retrieval event and message consumption event, and extract the message unique identifier, queue or topic name and timestamp. The kernel buffer module is used to temporarily store the event data captured by the eBPF probe module in the form of key-value pairs, where the key is the unique identifier of the message and the value includes the event type, queue or topic name and timestamp information. The user-space diagnostic engine module, running in user space, includes a data acquisition unit and a correlation analysis unit. The data acquisition unit continuously reads event data from the kernel buffer module through the interface provided by eBPF and sends the event data to the correlation analysis unit. The correlation analysis unit maintains message production event cache, message placement event cache, message retrieval event cache, and message consumption event cache, as well as a message status table. It uses the unique identifier of the message as an index to associate and match the production, placement, retrieval, and consumption events of the same message. Based on the matching results, it diagnoses the message delivery status and calculates the end-to-end delivery delay and segmentation delay. The output module is used to push diagnostic results and latency indicators to the unified monitoring system for visualization and alarms.

[0014] Compared with the prior art, the present invention has the following beneficial effects: (i) Collaboration between probe deployment and business semantic extraction Existing technologies mainly capture general data structures at the kernel level. Although these data can reflect the system status, they are not directly related to business messages, making it difficult for operations and maintenance personnel to infer the delivery status of specific business messages from kernel-level indicators.

[0015] This application analyzes the symbol tables of the message middleware client library and server program to accurately locate the offset positions of the unique message identifier and queue name in the parameter structures of message production, consumption, insertion, and retrieval functions. This design allows the raw data captured by the eBPF probe to directly carry business semantics, achieving a seamless connection from kernel-level capture to business-level understanding. The synergy between probe deployment and semantic extraction transforms the previously invisible "black box" processing within the message middleware into quantifiable and analyzable structured event data.

[0016] (ii) Synergy between Level 4 event capture and end-to-end reconstruction Existing technologies only capture two types of events: requests and responses, forming a two-stage tracing model of "caller → callee," which cannot reveal the details of message flow within the middleware.

[0017] This application captures production and consumption events on the client side, and insertion and retrieval events on the server side. These four levels of events cover the complete message delivery chain from producer to consumer. This design has the following synergistic effects: First, the four-level event structure breaks down the message processing, which is invisible within the message middleware, into observable atomic steps, allowing for precise location of any anomalies. Second, the collection of four-stage timestamps provides a data foundation for segmented delay calculation, enabling operations personnel to clearly distinguish whether the delay occurs on the production side, the server-side storage stage, or the consumption side. Finally, the capture of server-side insertion and retrieval events allows message backlog issues to be quantified as the difference between the number of insertion and retrieval events, achieving real-time perception of consumption lag from the client's perspective.

[0018] (III) Coordination between kernel-mode capture and user-mode memory matching In existing technologies, eBPF-based event capture and event analysis typically employ a separate architecture: captured event data is either written to disk and processed by an offline analysis system, or directly pushed to a remote monitoring platform for analysis. This architecture introduces a significant time window between data acquisition and analysis, making it difficult to meet real-time diagnostic needs.

[0019] This application adopts a two-level processing architecture of "kernel-mode capture + user-mode memory matching": the eBPF probe completes nanosecond-level capture of events in kernel mode and writes them to the eBPF mapping, while the user-mode process continuously reads events from the mapping through the interface provided by eBPF and maintains a four-level cache and message status table in memory for real-time matching. The synergy of this architecture is reflected in the clear division of responsibilities between kernel mode and user mode. Kernel mode is responsible for low-latency capture and minimal data filtering, while user mode is responsible for flexible association matching and status management. Data transmission between the two is completed through eBPF mapping, without going through disk or network, avoiding I / O overhead and serialization latency. User-mode memory matching uses the unique identifier of the message as a hash index, achieving O(1) lookup efficiency, so that matching and diagnosis can be completed within milliseconds as soon as the event arrives.

[0020] (iv) Dual diagnostic synergy of passive matching and active timeout Existing diagnostic mechanisms are mostly passive: monitoring systems display data, which is then manually compared and analyzed by operations and maintenance personnel, or alarms are triggered based on preset threshold rules for aggregated indicators. This approach makes it difficult to detect the loss or timeout of individual messages in real time. By the time operations and maintenance personnel discover an abnormal queue depth, messages may have already been lost for several minutes or even longer.

[0021] This application designs a dual diagnostic mechanism of "passive matching + active timeout": On the one hand, when a consumption event arrives, the system passively triggers a delivery success determination and calculates the insertion delay, storage / retrieval delay, retrieval delay, and total production / consumption delay based on the four timestamps recorded in the message status table; on the other hand, the system sets a timeout window for each record in the message status table, actively scanning and determining messages that have not matched a consumption event within the timeout period as delivery anomalies. This combined active and passive diagnostic paradigm achieves the following synergy: successful delivery of normal messages can be determined and delay statistics can be completed at the moment consumption occurs; loss or severe delay of abnormal messages can be actively detected when the timeout window expires; the length of the timeout window is configurable, enabling the system to adapt to the differentiated requirements of latency sensitivity in different business scenarios; the diagnostic results include the unique message identifier, the target queue name, and the timestamps of each stage, providing complete contextual information for fault location.

[0022] (v) Synergy between zero-intrusion implementation and financial-grade observability requirements Existing monitoring solutions often require the introduction of probes or SDKs into applications, or reliance on middleware to expose specific management interfaces. These solutions are all invasive to varying degrees to the original system. For core production systems in the financial industry, any code modification or component introduction must undergo a rigorous review and testing process, resulting in long implementation cycles and high risks.

[0023] This application's complete technical solution requires no modification to the message middleware code, no modification to the business application code, and no introduction of additional tracing components or message queues, achieving true zero-intrusion monitoring. The synergy of this feature is reflected in: eBPF probes are dynamically mounted on user-space functions of already running processes, without requiring restarting the target process or recompiling the code; the capture and transmission of event data are completed entirely within the operating system kernel and user-space diagnostic process, without relying on any external services; diagnostic results are pushed to a unified monitoring system via standard protocols, seamlessly integrating with the existing operations and maintenance system. This zero-intrusion characteristic allows this application to be quickly deployed in financial-grade production environments, while mitigating compliance risks and production hazards associated with code changes.

[0024] In summary, this application constructs a complete, efficient, and non-intrusive message delivery diagnostic solution through the organic synergy of technical features such as probe deployment, four-level event capture, kernel-mode data transmission, user-mode memory matching, and a dual diagnostic mechanism. The synergistic effect between these technical features significantly surpasses the sum of the individual features: the synergy between probe deployment and semantic extraction allows captured event data to be used for business diagnostics without secondary parsing; the synergy between four-level event capture and end-to-end reconstruction makes the black-box process of message delivery fully observable; the synergy between kernel-mode capture and user-mode memory matching achieves the performance goals of nanosecond-level capture and millisecond-level diagnosis; the synergy between passive matching and proactive timeout enables real-time confirmation of normal messages and proactive discovery of abnormal messages; and the synergy between zero-intrusion implementation and financial-grade observability requirements enables this application to be rapidly deployed in core production systems.

[0025] The aforementioned technical effects cannot be achieved simultaneously by any single solution in the existing technology. This application has achieved breakthrough improvements over the existing technology in multiple dimensions, including zero intrusion, message-level granularity, real-time diagnostic capabilities, end-to-end full-link coverage, and precise fault location. Attached Figure Description

[0026] Figure 1 : Overall flowchart of the method in Embodiment 1 of the present invention; Figure 2 : A schematic diagram of eBPF probe deployment and event capture according to an embodiment of the present invention; Figure 3 Flowchart of user-mode event association and diagnosis according to an embodiment of the present invention; Figure 4 : System architecture diagram of Embodiment 2 of the present invention. Detailed Implementation

[0027] To facilitate understanding of this invention, the following explanations are provided for the relevant terms used herein: eBPF (Extended Berkeley Packet Filter): A technology that allows user-defined programs to run securely and efficiently within the kernel. It enables the dynamic injection and execution of sandboxed programs without modifying the kernel source code or loading kernel modules, and can be used in scenarios such as networking, observability, and security.

[0028] uprobe: A dynamic probe mechanism provided by the Linux kernel, used to insert probes at function entry points or specified addresses in user-space applications, triggering eBPF program execution when the program reaches that location.

[0029] uretprobe: A probe mechanism used in conjunction with uprobe to trigger eBPF program execution when a user-mode function returns. It is typically used in conjunction with uprobe to capture context information before and after a function call.

[0030] eBPF Map: A key-value data structure used to exchange data between kernel mode and user mode. eBPF programs can write collected data into the map, and user-mode processes can read the contents of the map through a standard interface.

[0031] Message Unique Identifier (MsgId): A unique identifier assigned to each message by the message middleware, used to uniquely identify a message throughout its entire lifecycle.

[0032] End-to-end delivery delay: The total time elapsed from when the producer calls the message sending interface to send a message until the consumer successfully receives the message and completes consumption.

[0033] Segmented Delay: The entire message delivery chain is divided into several consecutive time periods according to the nodes of message flow, including the insertion delay from the producer to the server, the storage and retrieval delay within the server, and the retrieval delay from the server to the consumer.

[0034] Example 1 like Figure 1 As shown, this embodiment provides a message delivery diagnosis method for message middleware based on eBPF, using TongLink / Q message middleware version V8.1 as an example. Those skilled in the art should understand that this method is also applicable to other message middleware products with similar function call structures.

[0035] I. Probe Deployment Phase First, symbol table analysis is required for the target message middleware. The objdump tool is used to analyze the client dynamic link library libtl_tcapi.so used for message production and consumption, and the server-side runtime tl_clibroker used for message storage and forwarding.

[0036] Symbol table resolution reveals that the `Tlq_PutMsg_Base` function, used for message production, and the `Tlq_GetMsg_Base` function, used for message consumption, are located in the client library `libtl_tcapi.so`. Analysis shows that the third parameter of both functions points to a message information structure, which stores key information such as the message's unique identifier and the target queue name.

[0037] Meanwhile, symbol table resolution located the WriteMsgDataFromeBaseQue function (used for message placement) and the ReadMsgDataFromeBaseQue function (used for message retrieval) in the server program tl_clibroker. Analysis revealed that the second parameter of both functions points to a message information structure, which also contains a unique message identifier and the name of the target queue.

[0038] In the host operating system kernel running the message middleware client (including producers and consumers) and the server, a pre-compiled eBPF program is loaded. Specifically, the clang compilation command is run to compile the source file of the eBPF program (such as tlq_trace.c) into eBPF bytecode (such as tlq_trace.o); then the load_bpf_file function is called to load the eBPF binary file into the kernel, which verifies it and compiles it into the corresponding machine code; after executing the machine code, the eBPF program becomes effective in the kernel. This eBPF program is dynamically mounted to the entry points of the Tlq_PutMsg_Base and Tlq_GetMsg_Base functions in the message middleware client core library libtl_tcapi.so via uprobe, and simultaneously mounted to the return points of these two functions via uretprobe. In addition, the eBPF program is also mounted to the entry points of the WriteMsgDataFromeBaseQue and ReadMsgDataFromeBaseQue functions in the server-side agent tl_clibroker via uprobe, and to their return points via uretprobe.

[0039] It should be noted that, to ensure the stable operation of the business system, this embodiment also includes a degradation handling mechanism for probe loading failures. When an eBPF probe fails to load (e.g., due to kernel version incompatibility, insufficient permissions, or resource limitations), the probe deployment process will record detailed error logs and exit, while ensuring that the normal operation of the original message middleware business processes is not affected and business traffic is not interrupted. The unified monitoring system will capture the loading failure event and generate an alarm to notify the operations and maintenance personnel for handling.

[0040] II. Data Capture Phase like Figure 2As shown, when the producer application calls the message sending interface, the underlying layer triggers a call to the `Tlq_PutMsg_Base` function. At this time, the uprobe probe attached to the function's entry point is triggered, capturing the context information of this call, obtaining the function's third parameter (i.e., a pointer to the message information structure), and temporarily storing it in the eBPF mapping. When the function finishes execution and returns, the corresponding `uretprobe` probe is triggered, using the pointer address temporarily stored by the entry point probe to extract the message's unique identifier and the target queue or topic name from the message information structure. Specifically, the message's unique identifier is extracted from the twentieth member of the structure, and the target queue or topic name is extracted from the fourteenth member. Simultaneously, a current system timestamp is generated as the message's production time, denoted as T1, and the event type is marked as `Produce`.

[0041] After a message is sent to the server by the producer, the server program calls the `WriteMsgDataFromeBaseQue` function to write the message to the queue. At this time, the uprobe probe attached to the function's entry point is triggered, extracting the message's unique identifier and the target queue or topic name from the message information structure pointed to by the function's second parameter. The extraction locations are the same as before: the message's unique identifier is extracted from the twentieth member of the structure, and the target queue or topic name is extracted from the fourteenth member. Simultaneously, a current system timestamp is generated as the message's enqueue time, denoted as T2, and the event type is marked as Put.

[0042] When a consumer application initiates a message consumption request, the server program calls the `ReadMsgDataFromeBaseQue` function to retrieve a message from the queue. At this time, the uprobe probe attached to the function's entry point is triggered, extracting the message's unique identifier and target queue or topic name from the message information structure pointed to by the function's second parameter (the extraction location is the same as above). Simultaneously, a current system timestamp is generated as the message's dequeue time, denoted as T3, and the event type is marked as Get.

[0043] After the server returns the message to the consumer application, the consumer calls the `Tlq_GetMsg_Base` function to complete message reception. At this point, the uprobe probe attached to the function's entry point is triggered, retrieving the function's third parameter and temporarily storing it in the eBPF mapping. When the function finishes execution and returns, the corresponding `uretprobe` probe is triggered, extracting the message's unique identifier and target queue or topic name from the message information structure (at the same location as above). Simultaneously, a current system timestamp is generated as the message consumption completion time, denoted as T4, and the event type is marked as `Consume`.

[0044] The eBPF program described above writes each captured event message into the eBPF mapping in the kernel in real time as key-value pairs. This embodiment uses a ring buffer mapping of type BPF_MAP_TYPE_RINGBUF to achieve high-throughput, low-latency data transmission. The key is a unique identifier for the message, and the value includes information such as the target queue or topic name, event type, timestamp, and the process identifier of the event source. For Produce and Consume events, the value also includes the process identifier of the producer or consumer; for Put and Get events, the value also includes the server process identifier and the queue handle.

[0045] III. User-mode data collection phase The diagnostic daemon running in user space continuously reads event data from the aforementioned eBPF mapping through the interface provided by eBPF. Specifically, the diagnostic daemon calls the `perf_buffer_poll` function to listen for events arriving in the circular buffer, and immediately reads the data when new event data is written. After initially processing and formatting the raw event data, the diagnostic daemon sends the event data to the unified data association analysis daemon via a network communication protocol. The two daemons can communicate using the gRPC protocol to support cross-host deployment scenarios.

[0046] IV. Data Association Analysis Stage like Figure 3 As shown, the data correlation analysis daemon maintains four independent event caches: message production event cache, message insertion event cache, message retrieval event cache, and message consumption event cache. Each cache is implemented using a hash table structure, with the unique message identifier as the key and a structure containing the event type, timestamp, queue name, and cache creation time as the value. Each cache has a set data expiration time, which is 300 seconds by default. Expired records are periodically cleaned up by a background thread.

[0047] In addition, the data association analysis daemon also maintains a message status table, which also uses a hash table structure, with the message unique identifier as the key and a structure containing the following fields as the value: message unique identifier (string type), target queue or topic name (string type), production event type and production timestamp T1 (64-bit unsigned integer, storing nanosecond-level Unix timestamps), put event type and put into queue timestamp T2 (64-bit unsigned integer), take event type and take out queue timestamp T3 (64-bit unsigned integer), consume event type and consume timestamp T4 (64-bit unsigned integer), abnormal status flag (boolean type), timeout threshold (64-bit unsigned integer), and record creation time (64-bit unsigned integer).

[0048] The data correlation analysis daemon continuously receives event data from the diagnostic daemon. For each received event record, the event information is first stored in the corresponding cache according to its event type (Produce, Put, Get, or Consume), with the message's unique identifier as the index during storage.

[0049] Then, a lookup operation is performed in the message status table using the message's unique identifier as the key: If the record corresponding to the identifier does not exist in the message status table, a new record is created in the message status table. The event type and timestamp of the current event are stored in the corresponding fields, and the other event type and timestamp fields are set to empty. At the same time, the timeout timestamp field is calculated and set according to the preset timeout window, and the record creation time is set to the current system time.

[0050] If a record corresponding to this identifier already exists in the message status table, then update the corresponding timestamp field in the record according to the event type of the current event.

[0051] When the received event type is Consume, it indicates that the message has been successfully received and processed by the consumer. At this time, the data correlation analysis daemon determines that the message delivery was successful and calculates various delivery latency metrics based on the production time T1, queue entry time T2, queue retrieval time T3, and consumption time T4 recorded in the message status table. Specifically, the entry latency L1 equals T2 minus T1, representing the time consumed from message production to being written to the server queue; the retrieval latency L2 equals T3 minus T2, representing the time the message waits in the server queue to be consumed; the retrieval latency L3 equals T4 minus T3, representing the time consumed from the message being retrieved from the server to the consumer completing reception; and the total production-consumption latency L4 equals T4 minus T1, representing the total end-to-end time from message production to consumption completion.

[0052] The data correlation analysis daemon assembles the aforementioned successful delivery records and various latency indicators into a diagnostic result message. A successful delivery record includes a unique message identifier, target queue or topic name, production time T1, queue entry time T2, queue retrieval time T3, consumption time T4, and the calculated latency values ​​L1, L2, L3, and L4. Simultaneously, it checks if any fields in the message status table corresponding to other event types (excluding consumption events) are empty. If any field in production time T1, queue entry time T2, or queue retrieval time T3 is empty, it indicates that the message was not properly recorded at the corresponding stage. In this case, a "Delivery Node Abnormality" alarm record is generated and reported to the unified monitoring system.

[0053] V. Message Delivery Status Diagnosis and Delay Calculation The data correlation analysis daemon periodically (in this embodiment, it is set to perform a scan once per second) scans the message status table to check for message records that have timed out and have not matched consumption events. Specifically, for each record in the message status table, if the current system time has exceeded the preset timeout timestamp of the record and the consumption timestamp T4 field is still empty, it is determined that the message may be lost or severely delayed.

[0054] At this point, the data correlation analysis daemon generates a "delivery anomaly" alarm record. This record includes the message's unique identifier, the target queue or topic name, the production time T1 (if it exists), the queue entry time T2 (if it exists), the queue retrieval time T3 (if it exists), and sets the consumption time T4 to null to indicate the anomaly. This alarm record is then pushed to the unified monitoring system.

[0055] For successfully matched messages (i.e., recorded consumption events), the data correlation analysis daemon, in addition to generating a successful delivery record, also performs statistical analysis on latency metrics. Specifically, for each target queue, the daemon maintains a latency statistics data structure for messages in that queue in memory, including message count, total latency, and sum of squared latency, used to calculate the average latency. Simultaneously, quantile estimation algorithms (such as t-digest or histogram bucketing) are used to calculate P95 and P99 latency values ​​to measure the queue's service quality. The statistical period is configurable; in this embodiment, it is set to 60 seconds. The statistical results are pushed to the unified monitoring system every 60 seconds, after which the statistical counter is reset.

[0056] Regarding consumption lag analysis, the data correlation analysis daemon judges backlog in two ways: First, it counts the number of messages in the message status table whose production events have been recorded but whose consumption events have not been recorded. This number reflects the total number of messages that have not been consumed in a timely manner from the client's perspective. Second, it compares the difference between the number of put events and the number of take-out events on the server within the same time window. If the number of put events is consistently greater than the number of take-out events, it indicates that there is a message backlog in the server queue.

[0057] To ensure the accuracy of timestamps, this embodiment requires all host machines deploying eBPF probes to be configured with Network Time Protocol (NTP) service, ensuring that the system time of each host machine remains synchronized, with the time synchronization error controlled within 10 milliseconds. The diagnostic daemon on each host machine uses the local system time when collecting timestamps, and the time synchronization mechanism ensures the comparability of T1 to T4 timestamps across host machines.

[0058] For data cleanup, each record in the four event caches and message status tables has a creation time field. The background cleanup thread of the data correlation analysis daemon performs a cleanup operation every 60 seconds, traversing each cache and status table to delete records whose creation time exceeds the preset retention time (set to 300 seconds in this embodiment) to prevent unlimited memory growth.

[0059] VI. Diagnostic Result Output and Visualization The data correlation analysis daemon pushes the generated delivery success records, delivery anomaly alarms, delivery node anomaly alarms, and delay statistics to the unified monitoring system via HTTP or gRPC protocols.

[0060] After receiving the above diagnostic data, the unified monitoring system provides the following visualization functions: a real-time message delivery status dashboard, displaying recently received successful delivery records and abnormal alarms in list or card format; a queue health dashboard, displaying the message backlog, average latency, P95 latency, and P99 latency of each queue in dashboard format; a topology diagram, graphically displaying the message flow relationship between producers, servers, and consumers, with abnormal nodes highlighted in red; and an alarm list, displaying historical alarm records, supporting filtering and retrieval by queue name, unique message identifier, and alarm type.

[0061] The unified monitoring system also integrates alarm notification functions. When an alarm for delivery anomaly or node anomaly is received, the system can trigger an alarm according to preset alarm rules (such as three consecutive delivery anomalies, a single message delay of more than 5 seconds, etc.) and notify the operation and maintenance personnel via email, WeChat, or SMS to achieve proactive fault response.

[0062] Example 2 like Figure 4 As shown, this embodiment provides a message delivery diagnostic system based on eBPF message middleware, used to implement the method described in Embodiment 1. The system adopts a distributed deployment architecture, consisting of local components deployed on each host machine and a centralized unified monitoring system.

[0063] (a) eBPF probe module The eBPF probe module is deployed in the operating system kernel that runs the message middleware client and server, and is responsible for capturing message production, insertion, retrieval and consumption events.

[0064] This module contains the following sub-units: The probe injection unit is used to inject pre-compiled eBPF bytecode into the kernel when the eBPF program is loaded, and to mount it to the target function through the uprobe and uretprobe mechanisms. The specific mounting method is as described in Example 1: it is mounted to the entry and return points of the Tlq_PutMsg_Base and Tlq_GetMsg_Base functions in libtl_tcapi.so, and the entry and return points of the WriteMsgDataFromeBaseQue and ReadMsgDataFromeBaseQue functions in tl_clibroker.

[0065] The data extraction unit is used to extract the unique message identifier and queue or topic name from the parameter structure of the target function when the probe is triggered. As described in Example 1, the Produce and Consume events are extracted from the structure pointed to by the third parameter, and the Put and Get events are extracted from the structure pointed to by the second parameter. The unique message identifier is located in the twentieth member of the structure, and the queue name is located in the fourteenth member of the structure.

[0066] The timestamp generation unit is used to obtain the current system's nanosecond-level timestamp each time the probe is triggered, as the time of the event occurrence.

[0067] (ii) Kernel buffer module The kernel buffer module is used to temporarily store event data captured by the eBPF probe module and provides a data transfer channel from kernel mode to user mode.

[0068] This module contains the following sub-units: The circular buffer unit is implemented using an eBPF mapping of type BPF_MAP_TYPE_RINGBUF. This circular buffer supports a multi-producer, single-consumer write mode, allowing eBPF probes on different CPU cores to write event data in parallel without locking, thereby reducing performance overhead.

[0069] The data encapsulation unit encapsulates captured event data into a unified key-value pair format, where the key is a unique message identifier and the value is a combined data structure containing the event type, queue name, timestamp, and process identifier. The encapsulated data is then written to the circular buffer unit.

[0070] (III) User-mode diagnostic engine module The user-space diagnostic engine module runs in user space and is responsible for reading event data from the kernel buffer module and performing correlation matching and status diagnosis. This module, along with the eBPF probe module and the kernel buffer module, is deployed on the same host machine.

[0071] This module contains the following sub-units: The data acquisition unit continuously reads event data from the ring buffer of the kernel buffer module through the user-space interface provided by eBPF. This unit creates an independent listening thread that calls the `perf_buffer_poll` function to block and wait for new events to arrive. When event data is available, it is immediately read and parsed, deserializing the binary data into structured internal event objects. This unit then sends the parsed event objects to the correlation analysis unit. The data acquisition unit and the correlation analysis unit communicate using a lock-free queue to avoid blocking during the data acquisition process.

[0072] The correlation analysis unit is used to maintain the four event caches and message status tables described in Example 1, and to perform event matching and status update operations. This unit receives event objects from the data acquisition unit and processes them according to the logic described in step five of Example 1: storing the event in the corresponding cache, searching for or creating a record in the message status table, updating the corresponding event type and timestamp, determining successful delivery and calculating the delay, determining timeout exceptions and generating alarms.

[0073] Specifically, the correlation analysis unit contains the following data structures: The production event cache is implemented using a hash table, where the key is a unique identifier for the message and the value is a structure containing the production timestamp T1, the queue name, and the creation time.

[0074] The event is placed in the event cache, which is implemented using a hash table. The key is the unique identifier of the message, and the value is a structure containing the timestamp T2, the queue name, and the creation time.

[0075] Retrieve the event cache using a hash table, where the key is the unique identifier of the message and the value is a structure containing the retrieval timestamp T3, the queue name, and the creation time.

[0076] The consumption event cache is implemented using a hash table, where the key is a unique identifier for the message and the value is a structure containing the consumption timestamp T4, the queue name, and the creation time.

[0077] The message status table is implemented using a hash table. The key is the unique identifier of the message, and the value is a structure containing the unique identifier of the message, the queue name, T1, T2, T3, T4, an exception status flag, a timeout timestamp, and the record creation time. The T1, T2, T3, and T4 fields are all 64-bit unsigned integers storing nanosecond-level Unix timestamps.

[0078] The correlation analysis unit also maintains a background scanning thread that, according to the timeout detection logic described in Example 1, periodically traverses the message status table to check whether the timeout timestamp of each record has expired and the consumption event has not yet arrived. For records that have not been consumed after timeout, this unit generates a delivery anomaly alarm record.

[0079] This unit simultaneously maintains the latency statistics data structure for each queue, calculates the average latency, P95 latency, and P99 latency in the manner described in Example 1, and outputs the statistical results to the output module according to the configured statistical period (e.g., 60 seconds).

[0080] (iv) Output module The output module is used to push the diagnostic results and latency indicators generated by the user-mode diagnostic engine module to the unified monitoring system.

[0081] This module contains the following sub-units: The results encapsulation unit is used to encapsulate diagnostic results and latency metrics into a unified data format. Successful delivery records are in JSON format, containing a unique message identifier, queue name, and T1, T2, T3, T4, L1, L2, L3, and L4 fields. Delivery anomaly alarms are in JSON format, containing a unique message identifier, queue name, timestamps of each recorded stage, and an anomaly type field (with values ​​of "timeout" or "missing"). Latency statistics are in JSON format, containing queue name, statistical period start time, total number of messages, average latency, P95 latency, and P99 latency fields.

[0082] The transmission unit is used to send encapsulated data to the receiving endpoint of the unified monitoring system via the HTTP protocol. The transmission unit supports configuring the target address, retry policy, and timeout period. In this embodiment, the transmission unit adopts an asynchronous sending mode, so the sending operation does not block the main flow of the diagnostic engine module; in the event of a sending failure, the transmission unit writes the failure record to a local log file and performs a maximum of three retries according to an exponential backoff policy.

[0083] (v) Unified monitoring system The unified monitoring system is independently deployed on the central server, responsible for receiving diagnostic data pushed by the output modules of each host machine, and providing visualization and alarm functions. The unified monitoring system works in conjunction with the eBPF probe module, kernel buffer module, user-space diagnostic engine module, and output module deployed on each host machine to form a complete message delivery diagnostic system.

[0084] This module contains the following sub-units: The data receiving unit provides an HTTP interface to receive diagnostic data from the output modules of various host machines. This unit employs a high-concurrency network framework, supporting the simultaneous reception of a large number of data push requests from host machines. Upon receiving the data, this unit performs data format validation and integrity checks, writing valid data into the time-series database and alarm database.

[0085] The time-series database unit, powered by the Prometheus time-series database engine, stores delivery success records and latency statistics. It is indexed by timestamps and supports aggregate queries by queue name and time range.

[0086] The alarm database unit uses a MySQL relational database to store delivery anomaly alarm records and delivery node anomaly alarm records, and supports persistent storage and historical query of alarm records.

[0087] The front-end display unit is a visual interface implemented using web technology. This unit includes the following views: a real-time message delivery status dashboard, displaying the latest successful delivery records and error alarms in a scrolling list; a queue health dashboard, showing the latency index trends of each queue with a line graph and the message backlog with a bar chart; a topology diagram, graphically displaying the message flow relationships between producer, server, and consumer nodes, with node colors indicating health status (green for normal, red for abnormal); and an alarm management interface, supporting the querying, filtering, confirmation, and processing of alarm records.

[0088] The alarm notification unit triggers notifications based on preset alarm rules. These rules include: a single message delay exceeding a threshold (e.g., 5 seconds), the number of delivery anomalies exceeding a threshold within a unit of time (e.g., 10 messages per minute), and consecutive delivery failures in a specific queue. When a rule is triggered, the alarm notification unit sends an alarm message to the configured recipients via email, WeChat Work robot, or SMS gateway. The alarm message includes the name of the abnormal queue, a unique message identifier, the anomaly type, and the time of occurrence.

[0089] Example 3 This embodiment uses a commercial bank's payment and settlement system as an example to illustrate the application of the technical solution of this application in a real business scenario.

[0090] The payment clearing system uses TongLink / Q messaging middleware as the transmission channel for payment instructions, processing approximately 2 million payment transactions daily, with a peak throughput of 30,000 messages per minute. Prior to implementing the technical solution described in this application, the bank's operations team had repeatedly encountered customer complaints about payment instructions being "sent but not yet credited." Because the delivery status of individual messages within the messaging middleware could not be tracked, each troubleshooting session took an average of 4 to 6 hours, severely impacting customer experience and business continuity.

[0091] The bank deployed the technical solution of this application in its production environment as follows: An eBPF probe module was loaded into the application server kernel running TongLink / Q producers and consumers, and the same eBPF probe module was loaded into the server kernel running the TongLink / Q server-side agent. Each server was configured with Network Time Protocol (NTP) service to ensure time synchronization. The user-space diagnostic engine module was deployed locally on each server, and the unified monitoring system was deployed in the bank's operations and maintenance center.

[0092] After deployment, the bank's operations and maintenance personnel monitored message delivery status in real time through the unified monitoring system's front-end interface. During a peak daily transaction period, the system detected that a payment instruction's consumption event had not arrived within the timeout window, immediately generating a "delivery anomaly" alarm. The alarm record included the message's unique identifier, the producer server identifier, the server identifier, and the production timestamp. Based on the alarm information and the message's unique identifier, the operations and maintenance personnel performed a correlation query in the log system and found that the message had been successfully sent on the producer side, but the server had not recorded the corresponding delivery event. This indicated that the fault occurred in the network transmission between the producer and the server. Further investigation revealed that the network card of the server hosting the fault was experiencing intermittent packet loss. After replacing the network card, the problem was resolved. The time from the occurrence of this fault to its location was approximately 8 minutes, a significant reduction compared to previous times.

[0093] In subsequent monthly statistics, the bank's operations team, using the latency statistics provided by the technical solution in this application, discovered that the P99 latency of a certain queue consistently exceeded 500 milliseconds, surpassing the 200 millisecond threshold stipulated in the service level agreement. By analyzing the queue's L2 latency metrics, the operations team found that the latency primarily occurred during the message queuing phase on the server. Further investigation revealed a performance bottleneck in the queue's consumer application, with the message consumption rate significantly lower than the production rate. After scaling up the consumer application, the P99 latency decreased to 120 milliseconds, meeting the service level agreement requirements.

[0094] This embodiment verifies the effectiveness of the technical solution of this application in the core payment scenario of the bank: the zero-intrusion deployment method avoids the risk of production system changes; the real-time anomaly alarm capability greatly shortens the fault location time; and the latency statistics provide data support for capacity planning and performance optimization.

[0095] Example 4 This embodiment uses a commercial bank's credit card transaction anti-fraud system as an example to illustrate the performance of the technical solution in high-throughput, low-latency scenarios.

[0096] The bank's credit card transaction anti-fraud system requires risk assessment for each transaction within 100 milliseconds, with message delivery latency (the time from when the transaction system sends a transaction message to when the anti-fraud system receives the message and begins processing) controlled within 5 milliseconds. The system's peak throughput reaches 100,000 transaction messages per minute. Because the anti-fraud system is extremely performance-sensitive, any additional monitoring overhead could affect the real-time nature of risk assessment, causing fraudulent transactions to occur within monitoring blind spots. Previously, the bank attempted to use log-based tracking, but the introduced performance overhead increased message delivery latency by an average of 8 milliseconds, exceeding the 5-millisecond budget, and the solution was ultimately abandoned.

[0097] The bank deployed the proposed technical solution on an ARMv8 architecture server configured with a 4-core CPU and 8 gigabytes of memory for performance verification. The verification scheme involved using a message load testing tool to send simulated transaction messages to the TongLink / Q middleware at a rate of 100,000 messages per minute. The CPU utilization and message delivery latency of the server were recorded under both conditions: without the eBPF probe and with the eBPF probe loaded.

[0098] The test results are as follows: Without the probe loaded, the average server CPU utilization was approximately 45%, and the average end-to-end message latency was 3.2 milliseconds. After loading the probe, the average server CPU utilization was approximately 46%, an increase of about 1%, and the average end-to-end message latency was 3.3 milliseconds, an increase of about 0.1 milliseconds. Furthermore, statistical analysis of the event data captured during the probe loading period showed an event matching rate of 100%, with no event loss detected. This latency increase is far below the anti-fraud system's 5-millisecond latency budget, meeting business requirements.

[0099] The verification results show that, under the premise of meeting the requirements of message delivery diagnostic function, the technical solution of this application has minimal impact on the performance of the host machine, the CPU utilization rate increases by about 1%, and the latency increase is within an acceptable range. It can meet the stringent requirements of message delivery monitoring in high-performance transaction scenarios such as bank credit card transaction anti-fraud systems.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It should be noted that the scope of protection of the present invention is not limited to the specific message middleware product version, specific function name, specific structure member offset, specific cache type, specific timeout window value, specific communication protocol, specific statistical period, and specific deployment architecture shown in the above embodiments. Those skilled in the art should understand that, based on the technical concept disclosed in this invention, the above specific implementation details can be adjusted, replaced, or optimized according to the needs of actual application scenarios. For example, adjusting the function mount point and parameter offset to adapt to different versions of message middleware, adopting other types of eBPF mapping data structures, adjusting the timeout window parameters to adapt to the latency sensitivity requirements of different business scenarios, adopting different user-mode and kernel-mode communication mechanisms, or redistributing and deploying the functions of some modules across different physical nodes, all of which do not depart from the spirit and essence of the technical solution of this invention.

[0101] Any modifications, equivalent substitutions, or improvements made to the above embodiments within the spirit and principles of this invention, as well as the application of the technical solutions of this invention to other types of message middleware products, should be included within the scope of protection defined by the claims of this invention. Furthermore, the scope of protection of this invention includes not only the technical solutions defined in the above method and system claims, but also embodiments such as computer program products for implementing the method, computer-readable storage media storing the computer program, and electronic devices executing the method; these embodiments are all within the scope of protection intended by this invention.

Claims

1. A message delivery diagnostic method based on eBPF message middleware, characterized in that, include: Deploying eBPF probes: In the operating system kernel running the message middleware client and server, the entry and return points of the message production and consumption functions in the client library, as well as the entry and return points of the message insertion and retrieval functions in the server program, are mounted to the client library via the uprobe mechanism. Event capture: The eBPF probe captures message production, insertion, retrieval, and consumption events, extracts the unique message identifier and queue name from the parameter structure of the function corresponding to each event, and generates a timestamp for each event; Data transfer: The captured event data is written into the eBPF mapping in the form of key-value pairs with the message unique identifier as the key and the event type and timestamp as the value, and then transferred from kernel space to user space; Association matching: Maintain a four-level event cache and message status table in user space. Store event records in the corresponding cache according to event type, and perform matching and status updates in the message status table using the unique message identifier as an index. Delivery Diagnosis: When a consumption event is received, the delivery is determined to be successful and the delay is calculated based on the production time, insertion time, retrieval time and consumption time in the message status table; if no consumption event is received within the preset timeout window, the delivery is determined to be abnormal.

2. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, It also includes statistical steps: taking queues as units, calculating the average delay, P95 delay, and P99 delay of message delivery within a preset time window, and calculating the number of messages in the message status table whose production events have been recorded but whose consumption events have not been recorded as a consumption lag indicator.

3. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, When extracting the message unique identifier and queue name, the offset position of the message unique identifier and queue name in the function parameter structure is determined by analyzing the symbol table of the message middleware client library and server program, and the extraction is performed based on the offset position.

4. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, The message status table uses the message's unique identifier as a hash index and records the production time, placement time, retrieval time, consumption time, and timeout timestamp. When a consumption event is received, the placement delay, access delay, retrieval delay, and end-to-end total delay are calculated. The placement delay is the placement time minus the production time, the access delay is the retrieval time minus the placement time, the retrieval delay is the consumption time minus the retrieval time, and the end-to-end total delay is the consumption time minus the production time.

5. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, The delivery diagnosis also includes: periodically scanning the message status table, and generating a delivery anomaly alarm for message records whose current time has exceeded the preset timeout timestamp and whose consumption timestamp is empty.

6. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, When calculating the delay in the delivery diagnosis, if there is an empty field in the production time, placement time, or retrieval time, an abnormal alarm for the delivery node is generated.

7. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, Deploying eBPF probes also includes degradation handling: when an eBPF probe fails to load, an error log is recorded and the probe deployment process is exited, while the original business processes of the message middleware are kept running normally.

8. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, Each host machine deploying the eBPF probe is configured with a network time protocol service to keep the system time of each host machine synchronized, with the synchronization error controlled within 10 milliseconds.

9. The message delivery diagnosis method based on eBPF message middleware according to claim 1, characterized in that, In the association matching process, event data is read from the eBPF mapping in user space through the data acquisition process, and the event data is sent to the association analysis process through a lock-free queue.

10. A message delivery diagnostic system based on eBPF message middleware, characterized in that, include: The eBPF probe module is deployed in the operating system kernel running the message middleware client and server. It is used to capture message production, placement, retrieval and consumption events by attaching to the entry and return points of message production, consumption, placement and retrieval functions through uprobe and uretprobe, and extracting the message's unique identifier, queue name and timestamp. The kernel buffer module is used to temporarily store the event data captured by the eBPF probe module in key-value pairs, with the message unique identifier as the key and the event type and timestamp as the value. The user-mode diagnostic engine module includes a data acquisition unit and a correlation analysis unit. The data acquisition unit is used to read event data from the kernel buffer module. The correlation analysis unit is used to maintain a four-level event cache and a message status table, using the unique identifier of the message as an index to match events at each stage of the same message, diagnose delivery status, and calculate latency. The output module is used to push diagnostic results and delay indicators to the unified monitoring system.

Images