A distributed service call SpanLog real-time aggregation and retrieval system

Abstract

This invention discloses a real-time aggregation and retrieval system for distributed service call SpanLog logs, comprising a SpanLog generation subsystem, a SpanLog acquisition and transmission subsystem, and a SpanLog aggregation and analysis subsystem. It generates structured SpanLog logs during service calls, collects SpanLogs from source nodes and transmits them to aggregation nodes, and receives, stores, and analyzes SpanLogs from various source nodes, constructing a joint index to support end-to-end correlation retrieval. Simultaneously, it eliminates reliance on third-party ETL tools, managing the entire lifecycle of SpanLogs based on existing service bus infrastructure, achieving end-to-end closed-loop processing from log generation, acquisition, transmission to aggregation and retrieval. This invention provides highly reliable, low-intrusion, and scalable data support for link tracing, performance profiling, and fault diagnosis in large-scale distributed microservice architectures, solving the problems of low efficiency in distributed service call log collection, data redundancy, and difficulty in link tracing in power grid dispatching systems.

Description

Technical Field

[0001] This invention relates to a power grid dispatching and control technology, and more particularly to a highly reliable real-time aggregation and retrieval system for distributed service call SpanLog logs based on a service bus. Background Technology

[0002] With the deepening of smart grid construction and the profound integration of information technology and the power industry, the power grid dispatch and control system is undergoing a profound transformation from a traditional monolithic architecture to a distributed microservice architecture. In existing next-generation dispatch systems, business logic is broken down into numerous fine-grained services. Under this complex distributed architecture, issuing a power grid dispatch command or making a real-time data query often requires multiple cross-machine and cross-network calls between multiple service nodes. While this end-to-end interaction mode improves the system's decoupling and scalability, it also makes the system's operating status unpredictable. To ensure the safe and stable operation of the large power grid and achieve millisecond-level fault location and accurate analysis of performance bottlenecks, efficient collection, correlation retrieval, statistics, and analysis of SpanLog generated during distributed service calls have become key technical requirements for ensuring the high availability of the dispatch system.

[0003] SpanLog is the core data carrier of distributed tracing. It records detailed information about a single remote call or operation, including key information such as start time, end time, Span ID, and Trace ID. Unlike traditional TraceLog, SpanLog focuses on a single unit of work. Each Span represents a specific remote call and is the basic data unit for building a complete request chain view.

[0004] Currently, in the general internet and IT operations and maintenance field, the mainstream technical solutions for handling such massive distributed logs generally adopt the ETL (Extract-Transform-Load) architecture, relying on third-party components such as Kafka and Elasticsearch. However, when facing the special and demanding application scenario of a power grid dispatching system production area, these solutions encounter the following insurmountable technical bottlenecks:

[0005] Strict security isolation and deployment restrictions: The production area of ​​the power grid dispatching system (usually referring to Security Zone I and Zone II) is a protected object of the national critical information infrastructure, adhering to the principles of "security zoning, dedicated network, horizontal isolation, and vertical authentication." Due to security precautions against unknown vulnerabilities, the production area has extremely stringent audit standards for third-party open-source components or commercial software. Mainstream ETL tools (such as Kafka and Elasticsearch) have large components and complex dependencies, and have historically been found to have high-risk security vulnerabilities (such as the Log4j2 vulnerability), making them extremely difficult to pass vulnerability scans and security assessments in the power industry. Furthermore, the production area strictly prohibits the arbitrary deployment of additional nodes; any solution requiring the mounting of an agent alongside the service is strictly prohibited due to increased attack surface, preventing mainstream solutions from being implemented at the source.

[0006] Strong constraints from communication protocols and network environment: General-purpose big data and log components typically communicate using more complex and advanced protocols such as HTTP / HTTPS and gRPC, relying on sophisticated DNS discovery and service governance mechanisms. However, within the power grid dispatching production area, inter-service communication is strictly controlled. Network policies usually only allow the use of the most basic and controllable TCP long-connection protocol, or custom protocols based on national cryptographic algorithms. Furthermore, the internal network of the production area typically cannot directly access the external network to retrieve dependencies, and there are strict physical or logical isolations between servers. This incompatibility of protocol stacks prevents existing HTTP-based log collection tools from effectively exchanging data with the service bus within the production area, and they are also unsuitable for the high-latency, low-bandwidth dispatching and control network environment.

[0007] Extremely high requirements for real-time performance and resource isolation: Power grid dispatching operations place near-stringent demands on the system's real-time performance and determinism. Dispatch control services typically run on servers with reserved resources and high loads, where CPU and memory resources must be prioritized for the computation of core business logic (such as SCADA data processing and AGC regulation). Traditional ETL architectures, especially their streaming computing engines and index building processes, often consume significant computing resources and I / O bandwidth. Deploying heavyweight acquisition and processing programs on production nodes can easily lead to resource contention, causing delays in the execution of core dispatching instructions and even affecting the safe and stable operation of the power grid.

[0008] The complexity of end-to-end log correlation retrieval: In a distributed service architecture, simply collecting log text from a single point is far from sufficient. Operations personnel urgently need end-to-end log reconstruction capabilities based on TraceID (trace identifier), that is, quickly retrieving the call trajectory, time distribution, and exception stack trace of a specific request across all related services from hundreds of millions of log entries. Existing solutions mostly use text format to store logs, lacking standardized SpanLog generation specifications, resulting in low log parsing efficiency, difficulty in field extraction, and complex link correlation analysis.

[0009] Lack of link tracing in dual-machine hot standby scenarios: Power grid dispatching systems generally adopt a dual-machine hot standby architecture. Existing solutions cannot effectively trace the service call links in the primary-standby switching scenario, making fault location difficult.

[0010] The specific needs of the analysis: Operations personnel require the rapid acquisition of granular metrics such as the number of successful calls, the number of failed calls, and the call duration for each service call within a specified timeframe. Existing general solutions, without dedicated middleware support, rely on simple text searches or regular expression matching, resulting in response times measured in minutes under massive datasets. This fails to meet the timeliness requirements for rapid fault recovery, hinders efficient real-time statistics and retrieval, severely impacts the efficiency of troubleshooting and performance optimization, and makes it difficult to efficiently resolve distributed correlation issues within massive logs.

[0011] In summary, existing mature distributed log processing solutions, due to their reliance on heavy third-party components, security vulnerabilities, protocol incompatibility, and high resource consumption, fundamentally conflict with the special operating environment of power grid dispatching production areas, which requires high security, high reliability, and low resource consumption. Therefore, the industry urgently needs a new method that can achieve highly reliable, low-intrusion, real-time aggregation and end-to-end retrieval of distributed service logs in production areas without deploying ETL tools, relying solely on the existing TCP service bus, to fill the current technological gap in the field of power dispatching system operation and maintenance monitoring. Summary of the Invention

[0012] Purpose of the invention: The purpose of this invention is to provide a highly reliable real-time aggregation and retrieval system for SpanLog logs of distributed service calls based on a service bus.

[0013] Technical solution: The distributed service call SpanLog real-time aggregation and retrieval system of the present invention includes:

[0014] SpanLog generation subsystem: Deployed on each service node, responsible for generating structured SpanLog logs during service calls, including SpanLog format definition and generation module, TraceID generation module, SpanID generation module, multi-mode request differential recording module, and dual-machine hot standby link tracing module;

[0015] SpanLog Acquisition and Upload Subsystem: Deployed on the source node, it is responsible for scanning and parsing SpanLog files, including a log scanning module, a log parsing module, a model encapsulation module, and a data upload module;

[0016] SpanLog aggregation and analysis subsystem: Deployed on the aggregation node, it is responsible for receiving and storing SpanLogs from various source nodes, including message receiving module, message verification module, cache synchronization module, composite index module, and retrieval and analysis module.

[0017] The SpanLog generation subsystem uses the SpanLog structure to construct a structured log format. The SpanLog log format includes timestamp, TraceID, ParentSpanID, SpanID, client or server identifier, key field group, and process name field.

[0018] The timestamp field records the precise time of SpanLog generation for time-series analysis and performance statistics. The TraceID field uses 128-bit UUID compression encoding, generated by combining the high-order bits of the timestamp and the node identifier, and runs throughout the entire request chain, associating SpanLogs scattered across various service nodes to generate a complete call chain view. The ParentSpanID field identifies the parent call of the current call, supporting the hierarchical expression of multi-level nested calls. The SpanID field generates a unique identifier by incrementing in the call order under the same TraceID, with each SpanID corresponding to one service call, and supports multi-level nested call tracing under the same request. The client or server identifier field identifies the generation location of the SpanLog. The key field group serializes key-value pairs, recording detailed information about the service call; the key field group differs for different request types. The process name field generates the process identifier for the SpanLog, thus identifying the log source.

[0019] The multi-mode request differentiation recording module includes synchronous request SpanLog, asynchronous request SpanLog, and server response SpanLog.

[0020] The synchronization request SpanLog is generated when the client initiates a synchronization service request, before the service bus interface returns a response;

[0021] The asynchronous request SpanLog is generated when the client initiates an asynchronous service request, before the service bus interface returns a response;

[0022] The server-side response SpanLog is generated after the server completes the execution of the corresponding service.

[0023] The dual-machine hot standby link tracing module adopts a dual-channel parallel tracing and state awareness merging mechanism, which includes mirror dual-write marking, standby machine silent buffering strategy and status code driven connection.

[0024] The mirror double-write marker extends the primary / backup status M / S identifier in the CS identifier bit of the SpanLog. When a single call initiates a mirror call to both primary and backup nodes simultaneously, the system generates two Span records, corresponding to the call trajectories from primary node to primary node and from backup node to backup node, respectively.

[0025] The standby silent buffer strategy involves setting an identifier on the standby side to enter silent mode. In silent mode, the SpanLog generated by the standby is temporarily stored in a local circular buffer and not reported. When the primary node fails and triggers a primary-standby switchover, the standby will change the identifier and resend the SpanLog temporarily stored in the buffer.

[0026] The caller, driven by the status code, can perceive the dual-machine operating status in real time through the primary / backup status codes carried in the service bus heartbeat response.

[0027] The execution steps of the SpanLog generation subsystem include:

[0028] (11) The client calls the service bus interface and generates the corresponding type of SpanLog log according to the request mode;

[0029] (12) For synchronous requests, generate a SpanLog with the key field serviceRequestSync, and record the server IP, server port, request data volume, response data volume, time consumption, timeout, service handle, number of resends, function name, return value and sending model identifier;

[0030] (13) For asynchronous requests, generate a SpanLog with the key field AsyncserviceRequestSync to record the server IP, server port, time consumption, response data volume, return value and sending model identifier;

[0031] (14) After the server finishes execution, generate a SpanLog with the key field func, and record the function name, return value, client process name, client IP, requested data volume, response data volume and server execution time.

[0032] The execution steps of the SpanLog acquisition and uploading subsystem include:

[0033] (21) Periodically scan the specified directory to obtain SpanLog files. The SpanLog file scanning is divided into two modes: stock scanning and incremental scanning.

[0034] (22) Perform a full scan of existing files, and do not repeat the scan in subsequent scan cycles;

[0035] (23) Perform incremental scanning. Use file pointers to record the read position of the incremental file. Record the scanned file pointer during the first scan. Check whether the file hash code has changed during the second scan. If it has changed, it means that the log rotation has started and the pointer backtracking strategy is needed to read the complete log line.

[0036] (24) Perform log parsing, parse and check the legality of the scanned log files. The specific judgment of log legality is: whether each log line contains a preset log keyword. If it contains the keyword, it is judged as a legal log and parsing continues. If it does not contain the keyword, it is judged as an illegal log and discarded directly.

[0037] (25) Perform model encapsulation, and encapsulate the valid SpanLog row data into SpanLog objects according to the predefined standardized request or response format;

[0038] (26) Data management is divided into data statistics and data sending. Data statistics is to perform periodic statistical tasks, and to count the SpanLog request or response once every minute. Data sending is to encapsulate the minute-level SpanLog request or response into statistical messages in Protobuf format through the service bus and send them to the SpanLog aggregation node.

[0039] The execution steps of the SpanLog aggregation and analysis subsystem include:

[0040] (31) Verify the SpanLog statistical messages received from the service bus, parse the request or response type identifier in the message, divide the message data into two categories: client and server, and then perform data source validity verification and time validity verification; the validity verification extracts the source IP address of the message and determines whether the source IP address exists in the preset list of valid client IP addresses; the time validity verification parses the start timestamp and end timestamp in the message and determines whether they meet the preset time validity rules;

[0041] (32) For the verified SpanLog statistical messages, the cached data in memory is periodically scanned according to the configured synchronization period, and the scanned log data is migrated to the DM database for persistent storage using the batch write technology of DM database; the DM database is configured with four tables, which are respectively used to store the three types of logs recorded by the client and the one type of logs recorded by the server.

[0042] (33) Log data is partitioned and stored and its lifecycle is managed according to the time dimension.

[0043] In step (32), the retrieval results or statistical reports of the log data in the Dameng database are displayed visually through the console. The retrieval steps include:

[0044] (321) Construct a composite index of minute-level timestamp-service name-IP-port;

[0045] (322) Supports full-link association retrieval based on TraceID to restore the call trajectory of a specific request in each service node;

[0046] (323) The search parameters are in a four-segment format, with multiple parameter values ​​separated by commas and parameter fields and values ​​separated by equal signs. It supports conditional search and statistical analysis by service name, time range, and call result.

[0047] (324) The search results are displayed in a table format, including header information and table content information, wherein the header information includes process name and service name.

[0048] In step (33), the lifecycle management includes link coding embedding, management channel reuse, memory index replacement, and deterministic retrieval;

[0049] The link encoding is embedded in the encoding and decoding process and the interceptor is mounted. It tracks the context slot by extending the bytes of the existing packet header without adding any additional network connections.

[0050] The management channel multiplexing transmits SpanLog through the heartbeat channel, and a three-level priority queue is used to ensure tracking of low bandwidth without impacting production traffic.

[0051] The memory index is replaced by a TraceID hash table and a time-series window instead of the Elasticsearch inverted index, and the memory usage of the aggregation node is controlled within 512MB.

[0052] The deterministic retrieval provides precise TraceID routing and time-based binary scanning, sacrificing full-text search in exchange for predictable resource boundaries.

[0053] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) Standardized SpanLog generation mechanism: By constructing a structured log format through the "##" separator, the standardized generation of SpanLog is realized, which improves the efficiency of log parsing and the accuracy of field extraction; (2) Differentiated log recording strategy: Differentiated field recording strategies are designed for synchronous and asynchronous requests, which optimizes log storage efficiency while ensuring the integrity of key information; (3) Dual-machine hot standby link tracing: The innovative design of the SpanLog tracing mechanism in the dual-machine hot standby scenario ensures the integrity of the link during the primary-standby switch and solves the problem of link tracing difficulties in the fault switching scenario of the traditional solution; (4) Efficient collection and aggregation: Hybrid scanning The strategy and intelligent deduplication mechanism effectively solve the data collection problem in the log rotation scenario and avoid data duplication and loss; (5) accurate retrieval and analysis: the joint index and full-link association retrieval mechanism support millisecond-level link query, providing accurate data support for fault location and performance optimization; (6) high security level environment adaptation: without relying on third-party ETL tools, it only uses the existing service bus to realize the collection, aggregation and retrieval of SpanLog, which meets the deployment requirements of high security level environments such as power grid dispatching system; (7) clear subsystem boundary division: by dividing the system into three subsystems with clear boundaries, the responsibilities of each subsystem are clear and the interfaces are clear, which facilitates the development, testing, deployment and maintenance of the system and improves the scalability and maintainability of the system. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0055] Figure 2 This is a schematic diagram showing the boundary division of the three subsystems of the present invention;

[0056] Figure 3 This is a functional module structure diagram of the SpanLog generation subsystem of the present invention;

[0057] Figure 4 This is a structural diagram of the functional modules of the SpanLog acquisition and uploading subsystem of the present invention;

[0058] Figure 5 This is a functional module structure diagram of the SpanLog aggregation and analysis subsystem of the present invention;

[0059] Figure 6 This is a flowchart of the SpanLog generation process of the present invention;

[0060] Figure 7 This is a flowchart of the SpanLog data acquisition and upload process of the present invention;

[0061] Figure 8This is a flowchart of the SpanLog aggregation analysis process of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0063] Example 1

[0064] This embodiment provides a highly reliable real-time aggregation and retrieval method for distributed service calls (SpanLog) on ​​a service bus, addressing the difficulty of applying existing technologies in high-security environments such as power grid dispatching systems. The details are as follows:

[0065] (1) SpanLog structured format design

[0066] This method uses "##" as a separator to construct a structured log format. The SpanLog log format is defined as follows:

[0067] This format contains seven core fields:

[0068] Timestamp: A millisecond-level Unix timestamp that records the precise time of SpanLog generation for time series analysis and performance statistics.

[0069] TraceID: A globally unique identifier encoded in 16-bit hexadecimal format, or a 128-bit UUID compressed encoding, generated by combining the high-order bits of the timestamp and the node identifier. TraceID runs throughout the entire request chain, linking the SpanLogs scattered across various service nodes to form a complete call chain view.

[0070] ParentSpanID: Identifies the parent call of the current call, supporting the hierarchical relationship of multi-level nested calls. For the top-level call, the parent call ID is 0.

[0071] SpanID: A unique identifier generated in ascending order of call sequence under the same TraceID. Each SpanID corresponds to a specific service call. SpanID supports multi-level nested call tracing under the same request.

[0072] Client / Server Identifier: Identifies the location where the SpanLog is generated. "client" indicates that the SpanLog is generated by the client, and "server" indicates that the SpanLog is generated by the server.

[0073] Key field group: Serializes key-value pairs, starting with "key:keyword" followed by comma-separated key-value pairs, recording detailed information about service calls. The key field group varies for different request types.

[0074] Process name: The process identifier that generates the SpanLog, used to identify the source of the log.

[0075] The fields mentioned above are separated by ASCII code 0x23 ("##") to prevent accidental truncation of embedded data. This self-describing format enables log parsing that does not rely on the schema registry center, resolving the issue of log format parsing failure caused by the unavailability of external metadata services in high-security environments.

[0076] (2) Multi-mode request SpanLog differential recording

[0077] For both synchronous and asynchronous request invocation modes, a differentiated SpanLog field recording strategy was designed to optimize log storage efficiency while ensuring the integrity of critical information. This includes:

[0078] Synchronous Request SpanLog (serviceRequestSync): When a client initiates a synchronous service request, a SpanLog is generated before the service bus interface returns. Key fields include: server_ip (server IP address), server_port (server port number), reqlen (request data volume in bytes), reslen (response data volume in bytes), timeuse (service call timeout in milliseconds), timeout (service timeout in seconds), standby_flag (standby mode identifier, -1 indicates non-standby mode), handle (service handle number), resend (number of times automatic re-request is allowed on failure), func (function name), ret (return value, greater than or equal to 0 indicates success, less than 0 indicates failure), and flag (sending model identifier: 0 normal, 1 secure, 2 normal + standby, 3 secure + standby).

[0079] Asynchronous Request SpanLog (AsyncserviceRequestSync): When a client initiates an asynchronous service request, a SpanLog is generated before the service bus interface returns. Key fields include: server_ip (server IP address), server_port (server port number), timeuse (service call time in milliseconds), reslen (response data size in bytes), ret (return value), and flag (sending model identifier).

[0080] Server-side response SpanLog(func): SpanLog is generated after the server completes service execution. Key fields include: func (server-side function name), ret (return value), client_proc (client process name), client_ip (client IP address), reqlen (request data size in bytes), reslen (response data size in bytes), ptime (server-side execution time in milliseconds), and flag (sending model identifier).

[0081] (3) SpanLog tracing mechanism under dual-machine hot standby architecture

[0082] To address the issue that service call links are easily broken during primary / standby switching in the dual-machine hot standby mode of power dispatching systems, this invention designs a dual-channel parallel tracing and state-aware merging mechanism. Combined with the dual-machine hot standby SpanLog tracing mechanism, it uses multi-dimensional technical means to ensure the lossless migration and integrity of the link context during primary / standby switching.

[0083] The dual-channel parallel tracking and state-aware merging mechanism mainly includes three core technical methods:

[0084] One is the mirror double-write marker, which extends the primary / backup status M / S in the CS flag bit of the SpanLog. When a single call initiates a mirror call to both primary and backup nodes simultaneously, two Span records will be generated, corresponding to the call trajectories from primary node to primary node (CM→SM) and from backup node to backup node (CS→SS), respectively, providing basic data support for link continuity.

[0085] The second is the standby silent buffer strategy. The standby side is set to a silent mode with standby_flag=1. In this mode, the SpanLog generated by the standby will be temporarily stored in the local circular buffer and will not be reported. When the primary node fails and triggers the primary-standby switchover, the standby will batch switch the standby_flag flag to 0 and resend the temporarily stored SpanLog to ensure that the link data is not lost after the switchover.

[0086] Thirdly, status code-driven connection is used. The caller can perceive the dual-machine operation status in real time through the 2-bit primary / standby status code in the service bus heartbeat response. 00 represents the primary node is normal, 01 represents the primary node is abnormal, 10 represents the standby node being promoted to primary node, and 11 represents a dual-active state. The caller dynamically adjusts the ParentSpanID pointer according to this status code to ensure that the new primary node can correctly associate with the last primary node Span before the switch, avoiding link breakage.

[0087] Power grid dispatching systems commonly employ a dual-machine hot standby architecture to ensure high availability. The accompanying dual-machine hot standby SpanLog tracing mechanism further strengthens link integrity assurance. This mechanism adds a `standby_flag` field to the SpanLog to identify whether the current call is in standby mode: when `standby_flag` is -1, it indicates non-standby mode; when `standby_flag` is a non-negative integer, the value represents the specific standby machine number, clearly distinguishing between primary and standby call scenarios.

[0088] In a dual-machine hot standby scenario, the client sends requests to both the primary and standby servers simultaneously. SpanLog records the call information to each server, including their respective server_ip, server_port, and call results, forming a complete dual-node call data archive. Thanks to the consistency of TraceID, when the primary server fails and switches to the standby server, SpanLog can continuously trace the entire call chain using the unchanging TraceID, without interruption due to server switching. Operations personnel can use TraceID to associate all call records between the primary and standby servers, fully reconstructing the failover process and providing strong support for troubleshooting and system optimization.

[0089] (4) SpanLog acquisition and aggregation mechanism

[0090] The SpanLog acquisition and upload subsystem is deployed at the source node and is responsible for acquiring SpanLog data from the local node and uploading it to the aggregation node. The boundary of this subsystem is defined by the point from the SpanLog file to the service bus message transmission. This invention designs an efficient SpanLog acquisition and aggregation mechanism, specifically including:

[0091] Hybrid scanning strategy: Periodically poll a specified directory, performing a full scan of existing files in one go; for incremental files, use file pointers to record the read position to achieve efficient incremental loading. Existing scan involves performing a full scan and parsing of existing files in one go, and subsequent scan cycles will not repeat this scan; incremental scan uses file pointers to record the read position of incremental files. During the first scan, the scanned file pointer is recorded; during the second scan, it is checked whether the file's hash code has changed. If it has changed, it indicates log rotation, and a pointer backtracking strategy is needed to read the complete log line.

[0092] Intelligent deduplication and integrity assurance: When the system detects that a log file has been renamed due to rotation, it combines the information of the processed files recorded in the DM database to determine deduplication and adopts a pointer backtracking strategy (e.g., backtracking 2 lines from the last recorded position) to avoid incomplete log lines caused by the instant file renaming.

[0093] Validity check: Parse SpanLog line by line and check if it contains the preset log keyword "##key:". If it does, it is considered a valid log and parsing continues. If it does not, it is considered an invalid log and is discarded.

[0094] Distributed aggregation: The source node encapsulates the SpanLog into a self-developed Protobuf format message and transmits it to the aggregation node through the service bus. The Protobuf message uses binary encoding, which has a higher compression ratio and parsing efficiency compared to the text format.

[0095] Data Verification: The aggregation node performs data source validity verification and time validity verification on received messages to ensure data security. Receipt verification involves parsing the request / response type identifier in the SpanLog statistical messages received from the service bus, classifying the message data into client and server categories, and then performing data source validity verification and time validity verification. The validity verification includes: source IP address validity verification, which extracts the source IP address of the message and determines whether it exists in a preset list of valid client IP addresses; and time validity verification, which parses the start and end timestamps in the message and determines whether they meet preset time validity rules.

[0096] Cache synchronization and persistent storage: For verified SpanLog statistical messages, the cached data in memory is periodically scanned according to the configured synchronization cycle. Using DM database batch write technology, the scanned log data is migrated to the DM database for persistent storage, achieving hierarchical data storage. The DM database is configured with four tables, corresponding to three types of logs recorded by the client and one type of logs on the server. SpanLog storage management partitions and manages the lifecycle of log data according to the time dimension, including storing the current day's data in a designated partition and automatically cleaning up expired data according to the configured retention policy. Expired data refers to historical data earlier than the configured number of days, reducing database storage pressure.

[0097] Data management is divided into data statistics and data transmission. Data statistics involves executing periodic statistical tasks, which count the SpanLog request / response status every minute. Data transmission involves encapsulating minute-level SpanLog requests / responses into statistical messages in Protobuf format via the service bus and sending them to the SpanLog aggregation node.

[0098] To address the issues of duplicate collection and truncation loss that easily occur in SpanLog collection scenarios during log rotation, this invention designs a hybrid incremental and stock scanning method based on file state machine and boundary awareness. The core is a three-state scanning engine: First, the stock state: after the initial full scan, a file fingerprint index (composed of inode, device number, and first and last line hashes) is established and marked as FROZEN state. Subsequently, file system events are monitored through inotify, and a rescan is triggered only when an inode change or file size reduction is detected. Second, the incremental state: a dual-pointer structure is maintained for the ACTIVE state log—a persistent pointer (recording the physical offset of the last successful commit) and a pre-read pointer (the current read position). When the difference exceeds 4KB, batch commits are performed. Third, rotation boundary processing: when a file size reduction is detected (i.e., log rotation occurs), the hash codes of the last line of the old file and the first line of the new file are compared. If they do not match, the backtracking pointer is set to twice the average byte length of the last 10 lines, and line by line is matched to the continuation point with a consistent hash, ensuring that data is not lost or duplicated during log rotation.

[0099] (5) SpanLog retrieval and analysis mechanism

[0100] The SpanLog aggregation and analysis subsystem is deployed at the aggregation node and is responsible for receiving, storing, and analyzing SpanLogs from various source nodes. The boundary of this subsystem is from the reception of the service bus message to the output of the retrieval results. This invention constructs an efficient SpanLog retrieval and analysis mechanism, specifically including:

[0101] Composite Index Construction: Construct a composite index of "minute-level timestamp-service name-IP-port" in the DM database to support direct and quick location of call records of a specific service instance within a specific time window from distributed node log data.

[0102] End-to-end correlation retrieval: Supports end-to-end correlation retrieval based on TraceID, which can completely restore the call trajectory, response status and time cost of a specific request across various nodes.

[0103] Conditional search and statistical analysis: Supports conditional search by service name, time range, and call result, and provides statistics on service call success rate, failure rate, maximum, minimum, and average time consumption, etc.

[0104] To address the compliance constraints of prohibiting external middleware in Security Zone I of the power grid dispatching system, this invention designs a low-intrusion closed-loop architecture based on the native channel of the service bus, reusing the existing TCP service bus to achieve the entire lifecycle flow of tracking data.

[0105] (1) Link encoding embedded: The interceptor is mounted in the SDK encoding and decoding process, and the existing packet header is extended with an 8-byte trace context slot (TraceID+SpanID+CS flag) without adding any additional network connections;

[0106] (2) Management channel reuse: The existing heartbeat channel is used to transmit SpanLog, and a three-level priority queue (control > trace > service) is adopted to ensure low bandwidth for tracing and not impact production traffic;

[0107] (3) Memory index replacement: Replace the Elasticsearch inverted index with TraceID hash table + time series window, and control the memory usage of the aggregation node within 512MB;

[0108] (4) Deterministic retrieval: Provides TraceID precise routing (O(1) positioning) and time period binary scanning, giving up full-text retrieval in exchange for predictable resource boundaries.

[0109] Example 2

[0110] This embodiment provides a highly reliable real-time aggregation and retrieval system for distributed service calls (SpanLog) based on a service bus, specifically including:

[0111] The SpanLog generation subsystem is used to generate structured SpanLog logs during service calls. It includes a SpanLog format definition module, a TraceID / SpanID generation module, a multi-mode request recording module, and a dual-machine hot standby tracing module.

[0112] The SpanLog acquisition and upload subsystem is used to scan and parse SpanLog log files, including a log scanning module, a log parsing module, a model encapsulation module, a cache synchronization module, and a data upload module.

[0113] The SpanLog aggregation and analysis subsystem is used to transmit SpanLog to the aggregation node and store it persistently. It includes a message receiving module, a data verification module, a cache storage module, a composite index module, and a retrieval and analysis module.

[0114] The boundary definitions of each subsystem are shown in Table 1:

[0115] Table 1 Boundary Definitions of the Three Subsystems

[0116] SpanLog uses a structured format, dividing the log content into seven core fields using "##" separators: timestamp, TraceID, parent call ID, SpanID, client / server identifier, key field group, and process name.

[0117] The specific functional descriptions of each module in each subsystem are shown in Table 2:

[0118] Table 2 Functional Descriptions of Each Module in the Three Subsystems

[0119] Example 3

[0120] This embodiment combines Figure 1 , Figure 2 , Figure 3 and Figure 6 This document provides a detailed explanation of the functional module structure of the SpanLog generation subsystem and the SpanLog generation process.

[0121] like Figure 1 and Figure 2 As shown, the SpanLog generation subsystem is deployed on each service node, with its system boundary starting and ending at the point from service call initiation to SpanLog disk persistence. This subsystem comprises four core functional modules: the SpanLog format definition module, responsible for defining the seven-field structured log format; the TraceID / SpanID generation module, responsible for generating globally unique identifiers; the multi-mode request recording module, supporting differentiated recording of synchronous and asynchronous requests; and the dual-machine hot standby tracing module, supporting tracing in master-slave failover scenarios. The input to the SpanLog generation subsystem is service call request / response information, and the output is a structured SpanLog log file.

[0122] like Figure 3 As shown, the SpanLog generation subsystem adopts a layered architecture design, consisting of a format definition layer, an identifier generation layer, a request log layer, and a hot backup tracing layer from top to bottom. The SpanLog format definition module is located in... Figure 3 At the top level, a seven-field structured log format is constructed using "##" separators. The seven core fields are: timestamp (millisecond-level Unix timestamp), TraceID (16-bit hexadecimal globally unique identifier), parent call ID (identifies the parent call of the current call, 0 at the top level), SpanID (a unique identifier under the same TraceID, incrementing sequentially according to call order), client / server identifier ("client" or "server"), key field group (comma-separated key-value pairs starting with "key:keyword"), and process name (the name of the process generating the SpanLog). The TraceID / SpanID generation module is connected to the SpanLog format definition module. The TraceID uses 16-bit hexadecimal encoding, generated by combining the high-order part of the timestamp and the node identifier. The SpanID is generated sequentially according to call order under the same TraceID, starting from 1 and supporting the expression of multi-level nested call hierarchies. The multi-mode request logging module is located in... Figure 3In the middle section, differentiated field recording strategies are designed for synchronous and asynchronous request calling modes. The dual-machine hot standby tracing module is located in... Figure 3 At the underlying level, the standby mode is identified by the standby_flag field. When standby_flag is -1, it indicates non-standby mode, and when standby_flag is a non-negative integer, it indicates the standby number.

[0123] like Figure 6 As shown, the SpanLog generation process includes the following steps:

[0124] (1) The client calls the service bus interface;

[0125] (2) Determine whether it is a synchronous request. If it is a synchronous request, execute step (3); otherwise, execute step (4).

[0126] (3) Generate a synchronization request SpanLog, with the key field being serviceRequestSync, and record fields such as server_ip, server_port, reqlen, reslen, timeuse, timeout, standby_flag, handle, resend, func, ret, and flag;

[0127] (4) Generate asynchronous request SpanLog, with the key field being AsyncserviceRequestSync, and record fields such as server_ip, server_port, timeuse, reslen, ret, and flag;

[0128] (5) Determine whether it is a dual-machine hot standby mode. If it is, proceed to step (6); otherwise, proceed to step (7).

[0129] (6) Set the standby_flag flag to record the primary and standby server call information;

[0130] (7) The server executes the service call;

[0131] (8) The server generates a response SpanLog, with the key field being func, and records fields such as func, ret, client_proc, client_ip, reqlen, reslen, ptime, and flag;

[0132] (9) SpanLog logs are written to disk.

[0133] Taking the sca_measure_check service in the power grid dispatching system as an example, the following is an example of a synchronous request to SpanLog:

[0134] The SpanLog recorded a synchronization service request: timestamp was 1740630947937 milliseconds, TraceID was 11e6bf3e18797000, SpanID was 11e6bf3e18797001, the client sent an 11-byte request to the server with IP 192.10.12.158 and port 10091, the time taken was 0 milliseconds, the response data was 10 bytes, and the call was successful (ret=0).

[0135] An example of an asynchronous request using SpanLog is as follows:

[0136] The SpanLog recorded an asynchronous service request: timestamp was 1740630947887 milliseconds, TraceID was 11e6bf3e09387000, SpanID was 11e6bf3e09387001, the client sent a request to the server with IP 192.10.12.122 and port 11001, the time was 1 millisecond, the response data size was 2498605 bytes, and the call was successful (ret=1).

[0137] in, Figure 1 , Figure 2 , Figure 3 and Figure 6 The detailed explanation is as follows:

[0138] Figure 1The various subsystems interact via a service bus, forming an end-to-end closed-loop processing chain from log generation, collection and transmission to aggregation and retrieval. The SpanLog generation subsystem is deployed on each distributed service node. This subsystem covers the entire boundary from "service call initiation to SpanLog disk writing" and contains four core functional modules: the SpanLog format definition module, responsible for constructing a seven-field structured log format using "##" delimiters; the TraceID / SpanID generation module, responsible for generating a 16-digit hexadecimal globally unique identifier based on the high-order bits of the timestamp and the node identifier; the multi-mode request recording module, which designs differentiated field recording strategies for synchronous and asynchronous requests; and the dual-machine hot standby tracing module, which identifies the standby machine mode through the standby_flag field to ensure the integrity of the link during primary / standby failover scenarios. The input of this subsystem is service call request / response information, and the output is a structured SpanLog log file. The SpanLog collection and uploading subsystem is deployed on the source node. This subsystem covers the boundary of "SpanLog log file to service bus message transmission" and contains five pipelined functional modules: the log scanning module executes a hybrid stock / incremental scanning strategy, supporting file pointer recording and pointer rollback mechanisms; the log parsing module performs validity checks based on the "##key:" keyword; the model encapsulation module encapsulates valid log lines into standardized SpanLog objects; the cache synchronization module implements hierarchical storage of the memory buffer queue and the DM database; and the data upload module performs minute-level statistical tasks and sends Protobuf-formatted statistical messages to the aggregation node via the service bus. The SpanLog aggregation and analysis subsystem is deployed on the aggregation node. This subsystem covers the boundary from "service bus message reception to retrieval result output" and contains five modules forming a functional closed loop: the message receiving module is responsible for receiving Protobuf messages from the service bus; the message verification module performs dual verification of the source IP address's legality and time validity; the cache storage module uses DM database batch write technology to achieve persistent storage and supports time-based partitioned storage and lifecycle management; the composite index module constructs a "minute-level timestamp-service name-IP-port" composite index; and the retrieval analysis module supports full-link association retrieval based on TraceID as well as conditional retrieval and statistical analysis by service name, time range, and call result. Figure 1The data flow is represented by solid lines with arrows: First, the SpanLog generation subsystem generates structured logs and writes them to disk as log files during service calls; then, the SpanLog acquisition and upload subsystem scans, parses, encapsulates, and uploads data from local log files; each source node transmits Protobuf formatted messages to the aggregation node via the service bus; finally, the SpanLog aggregation and analysis subsystem completes data verification, persistent storage, and index construction, and outputs search results and statistical reports through the retrieval and analysis module. Overall, this invention demonstrates a technical solution that abandons third-party ETL tools and achieves full lifecycle management of SpanLog solely based on existing service bus infrastructure, meeting the compliance requirements and deployment constraints of high-security environments such as power grid dispatching systems.

[0139] Figure 2This diagram illustrates the boundary division of the three subsystems in this invention. The system architecture is divided into three clearly defined subsystems, which interact with each other through explicit interfaces. The SpanLog generation subsystem is deployed on each service node, with its system boundary defined from the initiation of a service call to the writing of the SpanLog to disk. This subsystem comprises four core functional modules: a SpanLog format definition module responsible for defining a seven-field structured log format; a TraceID / SpanID generation module responsible for generating globally unique identifiers; a multi-mode request recording module supporting differentiated recording of synchronous and asynchronous requests; and a dual-machine hot standby tracing module supporting link tracing in master-slave failover scenarios. The input to the SpanLog generation subsystem is service call request / response information, and the output is a structured SpanLog log file. The SpanLog acquisition and uploading subsystem is deployed on the source node, with its system boundary defined from the sending of the SpanLog log file to the service bus message. This subsystem comprises five core functional modules: a log scanning module that executes a hybrid scanning strategy for existing and incremental data; a log parsing module responsible for validating and parsing SpanLogs; a model encapsulation module that encapsulates valid SpanLogs into standardized objects; a cache synchronization module that synchronizes the memory cache with the local database; and a data upload module that encapsulates SpanLogs into Protobuf format messages and sends them via the service bus. The input to the SpanLog acquisition and upload subsystem is the SpanLog log file, and the output is a Protobuf format message. The SpanLog aggregation and analysis subsystem is deployed on the aggregation node, with its system boundary defined by the service bus message reception to the retrieval result output. This subsystem comprises five core functional modules: a message receiving module that receives Protobuf messages transmitted via the service bus; a message verification module that performs data source validity and time validity checks; a cache storage module that writes SpanLogs in batches to the DM database; a composite index module that constructs a "minute-level timestamp-service name-IP-port" composite index; and a retrieval and analysis module that supports end-to-end related retrieval and statistical analysis. The SpanLog aggregation and analysis subsystem takes Protobuf format messages as input and outputs search results and statistical reports. Figure 2 The data flow between the three subsystems is as follows: the structured log files generated by the SpanLog generation subsystem serve as the input source for the SpanLog acquisition and upload subsystem; the SpanLog acquisition and upload subsystem sends Protobuf format messages to the SpanLog aggregation and analysis subsystem via the service bus; the SpanLog aggregation and analysis subsystem completes the persistent storage and retrieval analysis of the data, and finally outputs the retrieval results.

[0140] Figure 3This diagram illustrates the functional module structure of the SpanLog generation subsystem in this invention. The SpanLog generation subsystem comprises four functional modules, each with clearly defined calling relationships and data flow logic. The SpanLog format definition module is responsible for defining the standardized format of SpanLog. This module uses "##" as a separator to construct a seven-field structured log format. The seven core fields are: timestamp (millisecond-level Unix timestamp), TraceID (16-bit hexadecimal globally unique identifier), parent call ID (identifying the parent call of the current call), SpanID (a unique identifier within the same TraceID, incrementing sequentially according to call order), client / server identifier ("client" or "server"), key field group (comma-separated key-value pairs starting with "key:keyword"), and process name (the name of the process generating the SpanLog). The TraceID / SpanID generation module is connected to the SpanLog format definition module and is responsible for generating globally unique call chain identifiers. The TraceID uses 16-bit hexadecimal encoding, generated by combining the high-order bits of the timestamp and the node identifier, ensuring global uniqueness in a distributed environment. SpanIDs are generated sequentially under the same TraceID, starting from 1 and incrementing in order of call. They support hierarchical representation of multi-level nested calls. The multi-mode request recording module designs differentiated field recording strategies for synchronous and asynchronous request modes. For synchronous requests (serviceRequestSync), key recorded fields include: server_ip (server IP address), server_port (server port number), reqlen (request data volume), reslen (response data volume), timeuse (service call duration), timeout (service timeout), standby_flag (standby mode identifier), handle (service handle number), resend (number of times automatic re-request is allowed on failure), func (function name), ret (return value), and flag (sending model identifier). For asynchronous requests (AsyncserviceRequestSync), key recorded fields include: server_ip, server_port, timeuses (service call duration), reslen, ret, and flag. The dual-machine hot standby tracking module uses the `standby_flag` field to identify the standby mode. When `standby_flag` is -1, it indicates non-standby mode; when `standby_flag` is a non-negative integer, it represents the standby machine number. In a dual-machine hot standby scenario, the client sends requests to both the primary and standby servers simultaneously. The `SpanLog` records the call information to both servers, ensuring the integrity of the connection during failover.

[0141] Figure 6 This is a flowchart of the SpanLog generation process in this invention. The SpanLog generation process includes the following steps:

[0142] (1) Client calls the service bus interface. When a service call is initiated, the system first determines the request type.

[0143] (2) Determine whether it is a synchronous request. If it is a synchronous request, proceed to step (3); if it is not a synchronous request (i.e., an asynchronous request), proceed to step (4).

[0144] (3) Generate a synchronization request SpanLog. The key field is serviceRequestSync, which records the server IP, server port, request data volume, response data volume, timeout, service handle, number of resends, function name, return value, and sending model identifier.

[0145] (4) Generate asynchronous request SpanLog. The key field is AsyncserviceRequestSync, which records the server IP, server port, time consumed, response data volume, return value, and sending model identifier.

[0146] (5) Determine whether it is a dual-machine hot standby mode. If it is a dual-machine hot standby mode, proceed to step (6); otherwise, proceed to step (7).

[0147] (6) Set the standby_flag flag. The standby_flag is set to a non-negative integer representing the standby server number, and at the same time records the call information to the primary server and the standby server.

[0148] (7) The server executes the service call. The server receives the request and executes the corresponding service logic.

[0149] (8) The server generates a response SpanLog. The key field is func, which records the function name, return value, client process name, client IP, requested data volume, response data volume, and server execution time.

[0150] (9) SpanLog writing to disk. The generated SpanLog is written to the log file in a structured format, completing the SpanLog generation process.

[0151] Example 4

[0152] This embodiment combines Figure 3 The dual-machine hot standby tracing module details the SpanLog tracing mechanism in dual-machine hot standby scenarios.

[0153] Power grid dispatching systems commonly employ a dual-machine hot standby architecture to ensure high availability. For example... Figure 3As shown, the dual-machine hot standby tracking module internally displays the values ​​for the `standby_flag` field: "-1 = non-standby mode", ">=0 = standby number". The right side of the module uses two parallel rectangles to represent the primary and standby servers, connected by an arc above them to indicate the failover relationship.

[0154] This invention designs a SpanLog tracing mechanism for dual-machine hot standby scenarios, ensuring link integrity during primary / standby failover through the following technical means: `standby_flag` field identification: A `standby_flag` field is added to the SpanLog to indicate whether the current call is in standby mode. When `standby_flag` is -1, it indicates non-standby mode; when `standby_flag` is a non-negative integer, it indicates the standby machine number. Dual IP concurrent request recording: In dual-machine hot standby scenarios, when the client sends requests to both the primary and standby servers simultaneously, the SpanLog records the call information to each server, including their respective `server_ip`, `server_port`, and call results. Link continuity guarantee: When the primary server fails and switches to the standby server, because the `TraceID` remains unchanged, the SpanLog can continuously trace the entire call link without interruption due to server switchover.

[0155] In a dual-machine hot standby scenario, the same service request will be sent to both the primary and standby servers simultaneously. The following two SpanLog entries record the same service calls to both servers:

[0156] As can be seen from the SpanLog above, the same service (sca_measure_check) simultaneously sends requests to both the primary server (192.10.12.122) and the backup server (192.10.12.152). The two requests use different TraceIDs and SpanIDs, but their timestamps are close. When the primary server fails, the system can seamlessly switch to the backup server, and the SpanLog fully records this process.

[0157] Example 5

[0158] This embodiment combines Figure 2 , Figure 4 and Figure 7 This document provides a detailed description of the functional module structure and data acquisition and transmission process of the SpanLog data acquisition and transmission subsystem.

[0159] like Figure 2As shown, the SpanLog acquisition and upload subsystem is deployed on the source node, with its system boundary defined by the transmission of SpanLog log files to the service bus. This subsystem comprises five core functional modules: a log scanning module that executes a hybrid existing / incremental scanning strategy; a log parsing module responsible for validating and parsing SpanLog entries; a model encapsulation module that encapsulates valid SpanLog entries into standardized objects; a cache synchronization module that synchronizes the memory cache with the local database; and a data upload module responsible for encapsulating SpanLog entries into Protobuf format messages and sending them via the service bus. The input to the SpanLog acquisition and upload subsystem is the SpanLog log file, and the output is a Protobuf format message.

[0160] like Figure 4 As shown, the SpanLog acquisition and upload subsystem adopts a pipeline architecture design, with each module executing sequentially from left to right. The log scanning module is represented by a magnifying glass icon, labeled "Hybrid Scanning Strategy" above, and divided into a stock scan area (labeled "One-Time Full Scan") and an incremental scan area (labeled "File Pointer Record Position") below. The right side of the incremental scan area is labeled with a description of the "Hash Code Check - Pointer Backtracking Strategy" processing flow. The log parsing module is represented by a gear icon, labeled "Legality Verification" above, with the verification rules marked in a judgment box shape: "Contains '##key:'? - Legal / Illegal". The model encapsulation module is represented by a cube icon, labeled "Standardized Encapsulation" above, with the encapsulated object structure listed in a table format: timestamp, TraceID, parent call ID, SpanID, client / server identifier, key field group, and process name. The cache synchronization module is represented by a database icon, labeled "Hierarchical Storage" above. Internally, it is divided into two areas: a memory buffer queue and a DM database. The right side lists the identifiers for four data tables: Client Synchronization Request Table, Client Asynchronous Request Table, Client Remote Request Table, and Server Response Table. The data upload module is represented by a send icon, labeled "Minute-Level Statistics" above. Internally, it is divided into a data statistics area (labeled "Statistics every minute") and a data sending area (labeled "Protobuf format encapsulation"). The right side of the module uses an envelope icon to represent messages sent via the service bus, labeled "Sent to Aggregator Node."

[0161] like Figure 7 As shown, the SpanLog acquisition and upload process includes the following steps:

[0162] (1) Start the log scanning task and scan the / var / spanlog directory;

[0163] (2) Determine if it is the first scan. If it is, proceed to step (3); otherwise, proceed to step (4).

[0164] (3) Perform a full scan of the existing data, record the file pointer position, and repeat the scan once.

[0165] (4) Perform incremental scanning, continue reading from the pointer position, and use hash code checking and pointer backoff strategies;

[0166] (5) Log validity check: check the keyword '##key:'. If valid, continue; otherwise, discard.

[0167] (6) Encapsulate the SpanLog model and extract information such as timestamp and TraceID;

[0168] (7) Data classification caching, stored in the memory buffer queue according to type (client synchronous request, client asynchronous request, client remote request, server response);

[0169] (8) Local data persistence, batch writing to the DM database;

[0170] (9) Determine whether the upload conditions (upload cycle + whitelist) are met. If they are met, proceed to step (10); otherwise, return to step (1) to continue scanning.

[0171] (10) Protobuf message encapsulation, minute-level statistical messages;

[0172] (11) Service bus sends data to the aggregation node.

[0173] in, Figure 4 and Figure 7 The detailed explanation is as follows:

[0174] Figure 4 This is a structural diagram of the functional modules of the SpanLog acquisition and upload subsystem in this invention. The SpanLog acquisition and upload subsystem contains five functional modules, which are executed sequentially in a pipeline manner. These include:

[0175] The log scanning module is the entry point for data acquisition. This module executes a hybrid scanning strategy, including both existing and incremental scanning modes. Existing scan performs a one-time full scan and parsing of existing files, and subsequent scan cycles do not repeat this process. Incremental scan uses file pointers to record the read positions of incremental files. During the first scan, the scanned file pointer is recorded; during the second scan, the file's hash code is checked for changes. If a change is found, it indicates log rotation, and a pointer backtracking strategy (dynamically calculated based on three times the average byte length of the most recent 10 lines) is used to read the complete log line.

[0176] The log parsing module is connected to the log scanning module and is responsible for parsing and validating the scanned log files. The validity criterion is: whether each line of log content contains the preset log keyword "##key:". If it does, the log is considered valid and continues parsing; otherwise, it is considered invalid and discarded.

[0177] The model encapsulation module encapsulates valid SpanLog line data into SpanLog objects according to a predefined standardized request / response format. The encapsulated object contains seven fields: timestamp, TraceID, parent call ID, SpanID, client / server identifier, key field group, and process name.

[0178] The cache synchronization module is connected to the model encapsulation module. It is responsible for storing the encapsulated SpanLog objects into a memory buffer queue according to their type, and periodically writing the data in the memory buffer queue into the DM database in batches. The DM database is configured with four tables, corresponding to client synchronous request logs, client asynchronous request logs, client remote request logs, and server response logs, respectively.

[0179] The data upload module is the data output module. This module performs periodic statistical tasks, compiling SpanLog request / response statistics every minute. It then encapsulates the minute-level SpanLog requests / responses into statistical messages in Protobuf format via the service bus and sends them to the SpanLog aggregation node.

[0180] Figure 7 This is a flowchart of the SpanLog acquisition and upload process in this invention. The SpanLog acquisition and upload process includes the following steps:

[0181] (1) Start the log scanning task. The source node starts a periodic log scanning task to scan the log files in the specified directory (such as / var / spanlog).

[0182] (2) Determine if it is the first scan. If it is the first scan, proceed to step (3); if it is not the first scan, proceed to step (4).

[0183] (3) Perform a full scan of the existing files. Perform a full scan and parsing of the existing files in one go, and record the scan status and file pointer position of each file.

[0184] (4) Perform incremental scan. Continue reading from the last recorded file pointer position, check if the file hash code has changed. If it has changed, it means that the log has rotated, and use the pointer backtracking strategy to read the complete log line.

[0185] (5) Log validity check. Parse SpanLog line by line and check if it contains the preset log keyword "##key:". If it contains it, it is determined to be a valid log and parsing continues. If it does not contain it, it is determined to be an invalid log and is discarded directly.

[0186] (6) SpanLog model encapsulation. Encapsulate the valid SpanLog row data into a SpanLog object according to a predefined standardized format, and extract information such as timestamp, TraceID, SpanID, and key field groups.

[0187] (7) Data classification and caching. The parsed SpanLog is classified and stored in the memory buffer queue according to its type (client synchronous request, client asynchronous request, client remote request, server response).

[0188] (8) Local data persistence. Periodically write SpanLog in the memory buffer queue to the DM database in batches to achieve hierarchical data storage.

[0189] (9) Determine whether the upload conditions are met. Determine whether the upload conditions are met based on the configured upload cycle and whitelist policy. If they are met, proceed to step (10); otherwise, return to step (1) to continue scanning.

[0190] (10) Protobuf message encapsulation. Extract SpanLog from the DM database and encapsulate it into a statistical message according to the Protobuf format.

[0191] (11) Service bus transmission. Protobuf format messages are sent to the SpanLog aggregation node via the service bus to complete the data collection and transmission process.

[0192] Example 6

[0193] This embodiment combines Figure 2 , Figure 5 and Figure 8 Provide a detailed description of the functional module structure and aggregation and storage process of the SpanLog aggregation and analysis subsystem.

[0194] like Figure 2As shown, the SpanLog aggregation and analysis subsystem is deployed at the aggregation node, with its system boundary spanning from the reception of service bus messages to the output of search results. This subsystem comprises five core functional modules: a message receiving module receives Protobuf messages transmitted via the service bus; a message verification module performs data source validity and time validity checks; a cache storage module writes SpanLogs in batches to the DM database; a composite index module constructs a composite index of "minute-level timestamp-service name-IP-port"; and a search and analysis module supports end-to-end related searches and statistical analysis. The input to the SpanLog aggregation and analysis subsystem is Protobuf format messages, and the output is search results and statistical reports.

[0195] like Figure 5 As shown, the SpanLog aggregation and analysis subsystem adopts a layered architecture design, consisting of a receiving layer, a verification layer, a storage layer, an indexing layer, and a retrieval layer from top to bottom. The message receiving module is represented by a receiving icon, labeled "Protobuf Message" above it. The received message format is represented by a binary data stream icon within the module. The message verification module is represented by a checking icon, labeled "Double Verification" above it. Internally, it is divided into a data source validity verification area (labeled "Source IP Address Whitelist Check") and a time validity verification area (labeled "Timestamp Validity Rules"). The verification result is "Verification Passed - Continue Processing" or "Verification Failed - Discard Message". The cache storage module is represented by a storage icon, labeled "Batch Write" above it. Internally, it is divided into two areas: a memory cache (labeled "Classified by Type") and a DM database (labeled "Partitioned Storage + Lifecycle Management"). The right side of the module is labeled "Automatically Clean Up Expired Data". The composite index module is represented by an index icon, labeled "Composite Index Construction" above. The module's internal structure uses a B+ tree icon, with index keys labeled "minute-level timestamp-service name-IP-port". A magnifying glass icon on the right side of the module indicates the index's ability to accelerate retrieval. The retrieval analysis module is represented by an analysis icon, labeled "Multi-dimensional Retrieval" above. Internally, it is divided into a TraceID full-link tracing area (labeled "Restore Call Trajectory"), a conditional search area (labeled "Service Name / Time Range / Call Result"), and a statistical analysis area (labeled "Success / Failure Count, Time Consumption Statistics"). A table icon on the right side of the module displays the search results, with table headers labeled "Process Name, Service Name".

[0196] like Figure 8 As shown, the SpanLog aggregation analysis process includes the following steps:

[0197] (101) Receive Protobuf messages from the service bus;

[0198] (102) Parse message types, which are divided into client / server types;

[0199] (103) Data source validity verification, source IP address whitelist check, continue if valid, discard if invalid;

[0200] (104) Time validity check, timestamp validity rules: if valid, continue; if invalid, discard.

[0201] (105) Data classification caching, stored in memory buffer queue according to type;

[0202] (106) Determine whether the synchronization period is met. If it is met, proceed to step (107); otherwise, return to step (101) to continue receiving messages.

[0203] (107) Batch writing to the DM database;

[0204] (108) Construct a composite index, with minute-level timestamp-service name-IP-port.

[0205] in, Figure 5 and Figure 8 The detailed explanation is as follows:

[0206] Figure 5 This is a structural diagram of the functional modules of the SpanLog aggregation and analysis subsystem in this invention. The SpanLog aggregation and analysis subsystem contains five functional modules, forming a complete processing link from message reception to retrieval output.

[0207] The message receiving module is responsible for receiving SpanLog statistical messages from the service bus. The received messages are encoded in Protobuf format, which has binary compression characteristics and has higher transmission and parsing efficiency compared to text format.

[0208] The message verification module is connected to the message receiving module and performs two types of verifications on the received messages: data source validity verification and time validity verification. Validity verification includes source IP address validity verification, which involves extracting the source IP address of the message and determining whether it exists in a preset list of valid client IP addresses; time validity verification involves parsing the start and end timestamps in the message and determining whether they meet preset time validity rules (e.g., the time difference does not exceed 24 hours).

[0209] The cache storage module is responsible for persistently storing verified SpanLog statistical messages according to the configured synchronization cycle. This module periodically scans the cached data in memory and uses batch write technology from the DM database to migrate the scanned log data to the DM database for persistent storage, achieving hierarchical data storage. Simultaneously, this module partitions and manages the lifecycle of log data according to time, storing the current day's data in a designated partition and automatically cleaning up expired data based on the configured retention policy.

[0210] The composite index module is connected to the cache storage module and is responsible for building a composite index of "minute-level timestamp-service name-IP-port" in the DM database. This composite index allows for direct and rapid location of call records of a specific service instance within a specific time window from distributed node log data, significantly improving retrieval efficiency.

[0211] The retrieval and analysis module is the system's output module. This module supports end-to-end correlation retrieval based on TraceID, capable of completely reconstructing the call trajectory, response status, and time cost of a specific request across various nodes. It supports conditional searches by service name, time range, and call result. It also supports statistical analysis of service call success count, failure count, maximum, minimum, and average call duration. Search parameters use a four-segment format, with multiple parameter values ​​separated by commas and parameter fields separated by equal signs. Search results are displayed in tabular form, with table headers including process name and service name.

[0212] Figure 8 This is a flowchart of the SpanLog aggregation analysis process in this invention. The SpanLog aggregation analysis process includes the following steps:

[0213] (101) Receive Protobuf messages. The aggregation node receives SpanLog statistics messages from the service bus.

[0214] (102) Parse message type. Parse the request / response type identifier in the message and divide the message data into two categories: client and server.

[0215] (103) Data source validity verification. Extract the source IP address of the packet and determine whether the source IP address exists in the preset list of valid client IP addresses. If valid, proceed to step (104); if invalid, discard the packet.

[0216] (104) Time validity check. Parse the start and end timestamps in the message and determine whether they meet the preset time validity rules. If valid, proceed to step (105); if invalid, discard the message.

[0217] (105) Data classification and caching. The verified SpanLogs are classified and stored in the memory buffer queue according to their type.

[0218] (106) Determine if the synchronization period is met. Determine if the batch write condition is met according to the configured synchronization period. If it is met, proceed to step (107); if it is not met, return to step (101) to continue receiving messages.

[0219] (107) Batch writing to DM database. The batch writing technology of DM database is used to migrate the SpanLog data in the memory cache to the DM database for persistent storage.

[0220] (108) Construct a composite index. Construct a composite index of "minute-level timestamp-service name-IP-port" in the DM database to support efficient retrieval.

[0221] (109) Determine whether a retrieval request has been received. If a retrieval request has been received, proceed to step (110); if not, return to step (101) to continue receiving messages.

[0222] (110) Perform search analysis. Perform conditional search or statistical analysis based on search parameters, supporting full-link association search based on TraceID and conditional search by service name / time range / call result.

[0223] (111) Output search results. Visualize the search results or statistical reports through the console. The search results are displayed in tabular form, including table header information and table content information, to complete the aggregate analysis process.

[0224] Example 7

[0225] This embodiment combines Figure 5 The search and analysis module and Figure 8 The retrieval process details the SpanLog retrieval and analysis mechanism.

[0226] like Figure 5 As shown, the search and analysis module is represented by an analysis icon, labeled "Multi-dimensional Search" at the top. The module is divided into three search function areas: the TraceID full-link tracing area (labeled "Restore Call Trajectory"), the conditional search area (labeled "Service Name / Time Range / Call Result"), and the statistical analysis area (labeled "Success / Failure Count, Time Consumption Statistics"). The right side of the module displays the search results in a table, with the table header labeled "Process Name, Service Name".

[0227] like Figure 8 As shown, the retrieval process includes the following steps:

[0228] (109) Determine whether a retrieval request has been received. If received, proceed to step (110); otherwise, return to step (101) to continue receiving messages.

[0229] (110) Perform retrieval analysis, full-link tracing / conditional retrieval / statistical analysis, supporting TraceID full-link tracing, service name retrieval, time range retrieval, and call result retrieval;

[0230] (111) Output the search results in a table format. The table header information includes the process name and service name.

[0231] The specific application scenarios for SpanLog retrieval and analysis are as follows:

[0232] Scenario 1: Full-link tracing based on TraceID. Operations personnel input a TraceID (e.g., "11e6bf3e18797000"), and the system retrieves all SpanLogs under that TraceID, displaying the call chain in chronological order, including the time and status of each stage such as client request, server processing, and response return.

[0233] Scenario 2: Service performance statistical analysis. Operations personnel input the service name (e.g., sca_measure_check) and a time range. The system then calculates metrics such as the number of calls, successful calls, failed calls, average execution time, maximum execution time, and minimum execution time for that service within the specified time range, displaying these metrics in tabular form.

[0234] Scenario 3: Fault Location and Analysis. When a service call fails, operations personnel can filter the failed SpanLog by the ret field and combine it with TraceID to trace the complete call chain, quickly locating the service node where the failure occurred and the specific cause.

Claims

1. A distributed service call SpanLog log real-time aggregation and retrieval system, characterized in that, include: SpanLog generation subsystem: Deployed on each service node, responsible for generating structured SpanLog logs during service calls, including SpanLog format definition and generation module, TraceID generation module, SpanID generation module, multi-mode request differential recording module, and dual-machine hot standby link tracing module; SpanLog Acquisition and Upload Subsystem: Deployed on the source node, it is responsible for scanning and parsing SpanLog files, including a log scanning module, a log parsing module, a model encapsulation module, and a data upload module; SpanLog aggregation and analysis subsystem: Deployed on the aggregation node, it is responsible for receiving and storing SpanLogs from various source nodes, including message receiving module, message verification module, cache synchronization module, composite index module, and retrieval and analysis module.

2. The real-time aggregation and retrieval system according to claim 1, characterized in that, The SpanLog generation subsystem uses the SpanLog structure to construct a structured log format. The SpanLog log format includes timestamp, TraceID, ParentSpanID, SpanID, client or server identifier, key field group, and process name field.

3. The real-time aggregation and retrieval system according to claim 1, characterized in that, The timestamp field records the precise time of SpanLog generation for time-series analysis and performance statistics. The TraceID field uses 128-bit UUID compression encoding, generated by combining the high-order bits of the timestamp and the node identifier, and runs throughout the entire request chain, associating SpanLogs scattered across various service nodes to generate a complete call chain view. The ParentSpanID field identifies the parent call of the current call, supporting the hierarchical expression of multi-level nested calls. The SpanID field generates a unique identifier by incrementing in the call order under the same TraceID, with each SpanID corresponding to one service call, and supports multi-level nested call tracing under the same request. The client or server identifier field identifies the generation location of the SpanLog. The key field group serializes key-value pairs, recording detailed information about the service call; the key field group differs for different request types. The process name field generates the process identifier for the SpanLog, thus identifying the log source.

4. The real-time aggregation and retrieval system according to claim 1, characterized in that, The multi-mode request differentiation recording module includes synchronous request SpanLog, asynchronous request SpanLog, and server response SpanLog. The synchronization request SpanLog is generated when the client initiates a synchronization service request, before the service bus interface returns a response; The asynchronous request SpanLog is generated when the client initiates an asynchronous service request, before the service bus interface returns a response; The server-side response SpanLog is generated after the server completes the execution of the corresponding service.

5. The real-time aggregation and retrieval system according to claim 1, characterized in that, The dual-machine hot standby link tracing module adopts a dual-channel parallel tracing and state awareness merging mechanism, which includes mirror dual-write marking, standby machine silent buffering strategy and status code driven connection. The mirror double-write marker extends the primary / backup status M / S identifier in the CS identifier bit of the SpanLog. When a single call initiates a mirror call to both primary and backup nodes simultaneously, the system generates two Span records, corresponding to the call trajectories from primary node to primary node and from backup node to backup node, respectively. The standby machine silent buffer strategy sets an indicator on the standby machine side to enter silent mode. In silent mode, the SpanLog generated by the standby machine is temporarily stored in the local circular buffer and is not reported. When the primary node fails and triggers a primary-standby switchover, the standby node will change its identifier and resend the SpanLog temporarily stored in the buffer. The caller, driven by the status code, can perceive the dual-machine operating status in real time through the primary / backup status codes carried in the service bus heartbeat response.

6. The real-time aggregation and retrieval system according to claim 1, characterized in that, The execution steps of the SpanLog generation subsystem include: (11) The client calls the service bus interface and generates the corresponding type of SpanLog log according to the request mode; (12) For synchronous requests, generate a SpanLog with the key field serviceRequestSync, and record the server IP, server port, request data volume, response data volume, time consumption, timeout, service handle, number of resends, function name, return value and sending model identifier; (13) For asynchronous requests, generate a SpanLog with the key field AsyncserviceRequestSync to record the server IP, server port, time consumption, response data volume, return value and sending model identifier; (14) After the server finishes execution, generate a SpanLog with the key field func, and record the function name, return value, client process name, client IP, requested data volume, response data volume and server execution time.

7. The real-time aggregation and retrieval system according to claim 1, characterized in that, The execution steps of the SpanLog acquisition and uploading subsystem include: (21) Periodically scan the specified directory to obtain SpanLog files. The SpanLog file scanning is divided into two modes: stock scanning and incremental scanning. (22) Perform a full scan of existing files, and do not repeat the scan in subsequent scan cycles; (23) Perform incremental scanning. Use file pointers to record the read position of the incremental file. Record the scanned file pointer during the first scan. Check whether the file hash code has changed during the second scan. If it has changed, it means that the log rotation has started and the pointer backtracking strategy is needed to read the complete log line. (24) Perform log parsing, parse and check the legality of the scanned log files. The specific judgment of log legality is: whether each log line contains a preset log keyword. If it contains the keyword, it is judged as a legal log and parsing continues. If it does not contain the keyword, it is judged as an illegal log and discarded directly. (25) Perform model encapsulation, and encapsulate the valid SpanLog row data into SpanLog objects according to the predefined standardized request or response format; (26) Data management is divided into data statistics and data sending. Data statistics is to perform periodic statistical tasks, and to count the SpanLog request or response once every minute. Data sending is to encapsulate the minute-level SpanLog request or response into statistical messages in Protobuf format through the service bus and send them to the SpanLog aggregation node.

8. The real-time aggregation and retrieval system according to claim 1, characterized in that, The execution steps of the SpanLog aggregation and analysis subsystem include: (31) Verify the SpanLog statistical messages received from the service bus, parse the request or response type identifier in the message, divide the message data into two categories: client and server, and then perform data source validity verification and time validity verification; the validity verification extracts the source IP address of the message and determines whether the source IP address exists in the preset list of valid client IP addresses; the time validity verification parses the start timestamp and end timestamp in the message and determines whether they meet the preset time validity rules; (32) For the verified SpanLog statistical messages, the cached data in memory is periodically scanned according to the configured synchronization period, and the scanned log data is migrated to the DM database for persistent storage using the batch write technology of DM database; the DM database is configured with four tables, which are respectively used to store the three types of logs recorded by the client and the one type of logs recorded by the server. (33) Log data is partitioned and stored and its lifecycle is managed according to the time dimension.

9. The real-time aggregation and retrieval system according to claim 8, characterized in that, In step (32), the retrieval results or statistical reports of the log data in the Dameng database are displayed visually through the console. The retrieval steps include: (321) Construct a composite index of minute-level timestamp-service name-IP-port; (322) Supports full-link association retrieval based on TraceID to restore the call trajectory of a specific request in each service node; (323) The search parameters are in a four-segment format, with multiple parameter values ​​separated by commas and parameter fields and values ​​separated by equal signs. It supports conditional search and statistical analysis by service name, time range, and call result. (324) The search results are displayed in a table format, including header information and table content information, wherein the header information includes process name and service name.

10. The real-time aggregation and retrieval system according to claim 8, characterized in that, In step (33), the lifecycle management includes link encoding embedding, management channel reuse, memory index replacement, and deterministic retrieval; the link encoding embedding is an interceptor mounted in the encoding and decoding process, which extends the byte tracking context slot in the existing packet header without adding extra network connections; the management channel reuse transmits SpanLog through the heartbeat channel, and uses a three-level priority queue to ensure low bandwidth tracking without impacting production traffic; the memory index replacement replaces the Elasticsearch inverted index with a TraceID hash table and a time-series window, and the memory usage of the aggregation node is controlled within 512MB; the deterministic retrieval provides TraceID precise routing and time-series binary scanning, giving up full-text search in exchange for predictable resource boundaries.

Images