Business identifier based multi-topic log link aggregation method, apparatus, device and medium

Abstract

The application provides a business identifier-based multi-topic log link aggregation method, device, equipment and medium, comprising: performing cross-table retrieval in parallel in a plurality of pre-constructed structured log topic tables based on a business identifier, and determining a plurality of target log entries in each structured log topic table that match the business identifier; merging the plurality of target log entries into a result set, globally ascendingly sorting the target log entries in the result set based on a timestamp field, and forming a log sequence arranged in chronological order; and generating a visual view of a transaction link based on the log sequence. The application solves the problem that the existing log system can only perform single-topic queries and cannot restore complete call links, and realizes automatic aggregation and display of logs scattered in multiple topics in chronological order in units of "one transaction", thereby improving system observability and troubleshooting efficiency.

Description

Technical Field

[0001] This application relates to the field of log processing technology, and in particular to a method, apparatus, device, and medium for multi-topic log link aggregation based on business identifiers. Background Technology

[0002] In modern distributed systems, a single user request (such as a payment transaction or order creation) typically spans multiple subsystems (such as gateways, authentication services, order services, payment services, and log services). Each system independently records its own logs and stores them in different log topics or log tables. Traditional log query systems usually support single-dimensional queries based on keywords, time ranges, or log topics, allowing users to view only log fragments from a specific service or topic. This model suffers from the following significant technical drawbacks: 1. Fragmented logs, unable to reconstruct the complete transaction chain. Existing systems only support querying and displaying logs within a single topic, failing to automatically correlate and aggregate multiple logs with the same `trace_id` generated in `gateway_log`, `auth_log`, and `order_log`. When facing faults, operations personnel must manually switch between multiple log interfaces, searching and comparing timestamps with business IDs (such as order numbers) one by one, which is time-consuming and prone to omissions. 2. Low troubleshooting efficiency and lack of root cause analysis support: When a request fails, traditional methods require manual assembly of log fragments, failing to automatically identify "which service timed out first," "which stage returned the error code," or "whether the error propagated upstream." 3. Invisible call order and performance bottlenecks: Logs contain timestamps, but their scattered storage prevents global sorting. Therefore, there is an urgent need for a technical solution that does not increase application intrusion, does not rely on dedicated tracing protocols, and can achieve automatic aggregation and visualization of call chains using only the structured fields of the logs themselves. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, apparatus, device and medium for multi-topic log link aggregation based on business identifiers, to solve the problem that existing log systems can only perform single-topic queries and cannot restore the complete call link, and to realize the automatic aggregation and display of logs scattered in multiple topics in chronological order based on "one transaction", thereby improving system observability and fault diagnosis efficiency.

[0004] This application provides a method for multi-topic log link aggregation based on business identifiers, the method comprising: Based on the business identifier, cross-table retrieval is performed in parallel in multiple pre-built structured log topic tables to determine multiple target log entries in each structured log topic table that match the business identifier. Multiple target log entries are merged into a result set, and the target log entries in the result set are sorted globally in ascending order based on the timestamp field to form a log sequence arranged in chronological order. A visual view of the transaction chain is generated based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

[0005] In one possible implementation, the business identifier based on user input is used to perform parallel cross-table searches in multiple pre-built structured log topic tables to determine target log entries in each structured log topic table that match the business identifier, including: A unified query syntax is constructed, and the business identifier is used as a query condition, which is synchronously distributed to all structured log topic tables; A combined index based on business identifier and timestamp is enabled for each structured log topic table. When the business identifier is detected to be missing in a certain structured log topic table, the system automatically downgrades to a fuzzy matching strategy to determine the log entries in each structured log topic table that match the business identifier.

[0006] In one possible implementation, the structured log topic table is constructed through the following steps: Configure a unified logging standard for each distributed subsystem; Based on the unified log specification, an independent log topic table is created for each of the distributed subsystems; wherein, the log topic table corresponds to a physically isolated data storage unit, each log topic table adopts a columnar storage structure, and the timestamp field and the business identifier field jointly establish a composite primary key or unique index; During the log collection phase, the log collector performs field standardization on the original logs and writes the standardized log entries into the corresponding log topic table, thus completing the construction of the structured log topic table.

[0007] In one possible implementation, after forming the log sequence arranged in chronological order, the multi-topic log link aggregation method further includes: For each target log entry in the log sequence, its structured fields are parsed. If the value of the field belongs to a preset error level enumeration set, the metadata of the target log entry is marked as an error level log. If the value of this field does not belong to the preset error level enumeration set, then perform error keyword regular expression matching on the original text content of the target log entry. If it matches the error keyword regular expression matching, then mark the metadata of the target log entry as an error level log. The total number of log entries marked as error level within a unit time window is counted. When the total number of log entries exceeds a preset alarm threshold, an alarm event is generated. The alarm event includes the name of the service that made the error, high-frequency error keywords, and associated business identifiers.

[0008] In one possible implementation, after generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: Based on the service name and time interval of adjacent target log entries, the call pattern between two adjacent target log entries is determined, and arrows are marked to indicate the call flow. Determine the time difference between adjacent target log entries. When the time difference exceeds a preset threshold, mark the corresponding service node with a high latency label. When any target log entry is identified as an error-level log, its service node is highlighted, and a red dashed line is added to connect it to its downstream service nodes.

[0009] In one possible implementation, after generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: In response to user commands to any service node, display the original log content and context log fragments of the target log entry corresponding to that service node.

[0010] In one possible implementation, the global ascending sort further includes: Before sorting, all timestamp fields are standardized to convert local timestamps from different distributed subsystems to UTC time and check whether their deviation exceeds the NTP synchronization tolerance threshold. If an out-of-range timestamp exists, the timestamp is marked as a suspicious time point, and a dashed border is displayed in the visualization view to prompt the user to verify the credibility of the target log entry.

[0011] This application embodiment also provides a multi-topic log link aggregation device based on a service identifier, the multi-topic log link aggregation device comprising: The cross-table retrieval module is used to perform cross-table retrieval in parallel in multiple pre-built structured log topic tables based on business identifiers, and to determine multiple target log entries in each structured log topic table that match the business identifiers. The global sorting module is used to merge multiple target log entries into a result set, and to sort the target log entries in the result set in global ascending order based on the timestamp field to form a log sequence arranged in chronological order. The view generation module is used to generate a visual view of the transaction chain based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

[0012] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the multi-topic log link aggregation method based on service identifiers described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the multi-topic log link aggregation method based on business identifiers as described above.

[0014] This application provides a method, apparatus, device, and medium for multi-topic log link aggregation based on business identifiers. The multi-topic log link aggregation method includes: performing parallel cross-table searches in multiple pre-built structured log topic tables based on business identifiers to determine multiple target log entries in each structured log topic table that match the business identifiers; merging the multiple target log entries into a result set; sorting the target log entries in the result set globally in ascending order based on a timestamp field to form a log sequence arranged in chronological order; and generating a visual view of the transaction link based on the log sequence. The visual view uses a time axis as the horizontal axis and service nodes as the vertical axis, with each target log entry mapped to a service node on the time axis with a status marker, the status marker including a log level identifier and an error keyword highlighting identifier. This solution addresses the limitation of existing log systems, which can only perform single-topic queries and cannot reconstruct the complete call chain. It automatically associates log entries from multiple log topics using a unified business identifier, sorts them globally by timestamp, and finally presents the complete call chain in a visual manner. This is the first time that a cross-system transaction process can be reconstructed using only log data. It solves the problem that traditional log systems cannot aggregate logs from multiple topics, and enables the automatic aggregation and display of logs scattered across multiple topics in chronological order, based on a "single transaction," thereby improving system observability and troubleshooting efficiency.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a multi-topic log link aggregation method based on business identifiers provided in this application embodiment; Figure 2 A flowchart illustrating the construction process of the structured log topic table provided in this application embodiment; Figure 3 This is one of the structural schematic diagrams of a multi-topic log link aggregation device based on a service identifier provided in an embodiment of this application; Figure 4 This is a second schematic diagram of a multi-topic log link aggregation device based on service identifiers provided in an embodiment of this application. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of log processing technology.

[0020] Research revealed the following shortcomings in existing technologies: 1. Fragmented logs, unable to reconstruct the complete transaction chain: Current systems only support querying and displaying logs within a single topic, failing to automatically associate and aggregate multiple logs with the same `trace_id` generated in `gateway_log`, `auth_log`, and `order_log`. When facing faults, operations personnel must manually switch between multiple log interfaces, searching and comparing timestamps with business IDs (such as order numbers), which is time-consuming and prone to omissions. 2. Low troubleshooting efficiency, lacking root cause localization support: When a request fails, traditional methods require manually piecing together log fragments, unable to automatically identify "which service timed out first," "which link returned an error code," or "whether the error propagated upstream." 3. Invisibility of call order and performance bottlenecks: Logs contain timestamps, but scattered storage prevents global sorting. Therefore, a technical solution is urgently needed that does not increase application intrusion, does not rely on dedicated tracing protocols, and only utilizes the structured fields of the logs themselves to achieve automatic aggregation and visualization of the transaction chain.

[0021] Based on this, this application provides a multi-topic log link aggregation method based on business identifiers, which solves the problem that existing log systems can only perform single-topic queries and cannot restore the complete call link. It realizes automatic aggregation and display of logs scattered in multiple topics in chronological order, taking "one transaction" as the unit, thereby improving system observability and fault diagnosis efficiency.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-topic log link aggregation method based on business identifiers, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the multi-topic log link aggregation method includes: S101: Based on the business identifier, perform cross-table retrieval in parallel on multiple pre-built structured log topic tables to determine multiple target log entries in each structured log topic table that match the business identifier.

[0023] In this step, cross-table searches are performed in parallel across multiple pre-built structured log topic tables based on the business identifier to identify multiple target log entries in each structured log topic table that match the business identifier.

[0024] Here, the logs output by each sub-service in the distributed subsystem are pre-mapped into independent structured log topic tables with a unified metadata structure according to the topic. Several key business context fields are forcibly defined and persisted in each table. Based on this, after receiving the business identifier input by the user, the system uses the business identifier as the query key to concurrently initiate standardized SQL queries in all registered structured log topic tables and synchronously retrieve matching entries.

[0025] In one possible implementation, the business identifier based on user input is used to perform parallel cross-table searches in multiple pre-built structured log topic tables to determine target log entries in each structured log topic table that match the business identifier, including: S1011: Construct a unified query syntax, using the business identifier as a query condition, and synchronously distribute it to all structured log topic tables.

[0026] Here, business identifiers are used as query conditions based on a unified query syntax to generate a dedicated execution plan for each target subject table, achieving syntax-level uniformity and execution-level isolation.

[0027] S1012: Enable a combined index based on business identifier and timestamp for each structured log topic table. When the business identifier is detected to be missing in a certain structured log topic table, automatically downgrade to a fuzzy matching strategy to determine the log entries in each structured log topic table that match the business identifier.

[0028] Here, all structured log topic tables are forcibly configured with the following indexing strategy during the table creation phase: Primary sort key: Utilizing the locally ordered characteristics of column-oriented storage engines such as ClickHouse, logs with the same `business_id` are distributed continuously in physical storage, greatly improving point query efficiency. Skip index: A `bloom_filter(256)` type index is created for the `business_id` field, and a `minmax` type index is created for the `timestamp` field, supporting millisecond-level filtering. Secondary index: For high-frequency fuzzy query scenarios, an additional Bloom filter index is created to quickly exclude partitions with no matching possibility. This composite index design keeps the average query response time of a single table ≤15ms and supports high-concurrency point queries with tens of millions of `business_id`s.

[0029] Specifically, a metadata probe is executed on each structured log topic table to verify in real time whether a declared business identifier field exists in the topic. If a missing field is detected, a degradation process is automatically activated, performing fuzzy matching in the following priority order: synonym field mapping matching, context regular expression extraction matching, and business context association inference, to determine the log entries in each structured log topic table that match the business identifier.

[0030] For further details, please refer to Figure 2 , Figure 2 A flowchart illustrating the construction process of the structured log topic table provided in this application embodiment. For example... Figure 2 As shown: S10121: Configure a unified logging standard for each distributed subsystem.

[0031] Here, the unified logging specification is an engineering standard document that defines the minimum structured contract that all distributed subsystems (such as gateway services, authentication centers, order engines, payment channels, risk control engines, etc.) must follow when outputting logs. Its core content includes: a set of required fields and a set of optional fields. The required fields include strings, unique service names, enumeration values, and at least one standardized business identifier field. The optional fields include fields that can be extended with `user_id`, `region`, `app_version`, `http_status`, etc., but type and length constraints must be declared. Log line format constraints and compliance verification mechanisms are also included. This specification is published to the enterprise internal knowledge base in the form of a YAML configuration file and synchronized to the configuration centers of each subsystem via GitOps to ensure full-stack consistency.

[0032] S10122: Based on the unified log specification, create an independent log topic table for each of the distributed subsystems; wherein, the log topic table corresponds to a physically isolated data storage unit, each log topic table adopts a columnar storage structure, and the timestamp field and the business identifier field jointly establish a composite primary key or unique index.

[0033] Here, based on the unified log specification, an independent log topic table is created one-to-one for each distributed subsystem, with each table corresponding to a physically isolated data storage unit. The specific process is as follows: In the unified log storage platform (such as a ClickHouse cluster), a dedicated database or dedicated table engine is allocated to each service to strictly map the table name to the service name `service_name`. All topic tables adopt a columnar storage structure, utilizing the columnar storage engine's extreme compression and vectorized calculation capabilities for high-frequency filtering fields such as timestamp and trace ID `trace_id`. The table structure strictly follows the ULS field definition. Specifically, `timestamp` and `trace_id` together form the sort key and primary key logical anchor. A TTL partitioning strategy is established on `(trace_id, timestamp)` to balance query efficiency and lifecycle management; a separate `bloom_filter(256)` hop count index is created for `trace_id`, and a `minmax` index is created for `timestamp` to achieve sub-millisecond point lookups. This design ensures that the various topic tables are completely decoupled at the physical level, avoiding cross-service log write conflicts, while also ensuring that logs with the same `trace_id` are locally ordered on the disk, laying the storage foundation for subsequent global sorting.

[0034] S10123: During the log collection phase, the log collector performs field standardization processing on the original logs, and writes the standardized log entries into the corresponding log topic table to complete the construction of the structured log topic table.

[0035] The standardization process includes format parsing, field completion, field normalization, and structured encapsulation.

[0036] Here, the standardized log entries are directly written by the collector to the corresponding service's log topic table via efficient protocols (such as HTTPPOST or KafkaProducer). (For example, the gateway log `gateway_v2_log` table only receives standardized logs from the gateway service.) The writing process ensures that the collector verifies whether the field names and types match the target table's DDL before writing. If they do not match, the write is rejected and an alert is issued. Furthermore, transactional commit is enabled for each batch of writes to prevent data inconsistency caused by partial writes.

[0037] S102: Merge multiple target log entries into a result set, and sort the target log entries in the result set in global ascending order based on the timestamp field to form a log sequence arranged in chronological order.

[0038] In this step, multiple target log entries are merged into a unified result set. Then, the target log entries in the result set are sorted globally in ascending order according to the timestamp field corresponding to each target log entry, forming a log sequence arranged in chronological order, which is used to completely reconstruct the call chain.

[0039] Here, the process of merging multiple target log entries into a unified result set is as follows: The system receives query responses from N structured log topic tables (N≥2), each response being a log entry. A min-heap-driven streaming architecture is adopted, with the following execution logic: Initialize a priority queue, using `timestamp` as the sorting key (ascending order), for each input log stream (N in total), pre-read the first log entry and push its tuple into the heap, executing the following loop: pop the top tuple from the heap → write the log entry pointed to by `record_ptr` to the output buffer (appending sequentially) → read the next log entry from the corresponding `stream_id` stream → if it exists, push the new tuple into the heap. When all N streams are exhausted, the merging ends, and the result set is obtained. The entire process does not maintain any cross-batch state, and memory usage remains constant.

[0040] In one possible implementation, the global ascending sort further includes: Before sorting, all timestamp fields are standardized by converting local timestamps from different distributed subsystems to UTC time and verifying whether their deviation exceeds the NTP synchronization tolerance threshold. If there is an out-of-tolerance timestamp, it is marked as a suspicious time point and prompted to check the credibility of the target log entry with a dashed border in the visualization view.

[0041] Here, all local timestamps are losslessly converted to Coordinated Universal Time (UTC) to generate standardized fields. For each standardized field, the instantaneous time difference δ between its UTC time and the current time synchronization with the Coordinated Universal Time (UTC) server is calculated. If δ > the preset NTP synchronization tolerance threshold, the log timestamp is determined to be an out-of-tolerance timestamp and marked as a suspicious time point.

[0042] In this application, as long as the structured logs output by each subsystem contain a unified business identifier field (such as `trace_id`, `order_id`), automatic association and link reconstruction of cross-topic logs can be completed without modifying business code or introducing additional SDKs or middleware.

[0043] S103: Generate a visual view of the transaction chain based on the log sequence; wherein the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

[0044] This step transforms the chronologically ordered log sequence into an interactive visual view with causal expression capabilities, anomaly sensitivity identification capabilities, and contextual traceability through four layers of technology: semantic parsing, temporal mapping, service topology binding, and dynamic rendering control. This method does not rely on static chart templates or hard-coded front-end rendering logic. Instead, it uses structured fields from the log sequence as the sole data source and drives front-end components to dynamically construct the visualization model through deterministic algorithms. This ensures that the view content is strictly consistent with the underlying log semantics, synchronized in real time, and verifiable and traceable.

[0045] In the specific implementation process, (1) Log sequence semantic parsing and node modeling: The system receives the log sequence after global time sorting, performs lightweight semantic annotation on each log, uses `service_name` as the unique key, automatically clusters all log entries of the same service, generates a logical service node object, and automatically identifies the service call transition based on the changes in the `merge_order` sequence number and `service_name` of the log entries to generate a directed edge. This process outputs a weighted directed graph data structure, where the number of nodes ≤ the total number of services and the number of edges ≤ the number of log entries. 1. Provide a topological skeleton for subsequent visualization. (2) Time axis-service topology dual coordinate mapping: The visualization engine adopts a two-dimensional coordinate system dynamic layout algorithm to map the above graph structure into visual elements. The horizontal axis is the absolute time axis: the first log `timestamp` in the sequence is taken as the origin `t0`, and the last log `timestamp` is taken as the endpoint `t1`, and scaled proportionally to the canvas width. Each log is positioned on the horizontal axis at its `(timestamp` `t0` corresponds to the pixel position. The vertical axis represents the service logic layer: based on the pre-defined service dependency topology map, the gateway layer (Layer 0), authentication layer (Layer 1), business orchestration layer (Layer 2), atomic service layer (Layer 3), etc., are arranged in layers with fixed vertical spacing, and services in the same layer are arranged horizontally side by side. Node rendering rules: each service node is drawn as a rounded rectangle, color-coded (green = success, red = failed, gray = missing), the width reflects the number of log entries for that service (reflecting participation), and the height is fixed. Edge rendering rules: call edges are drawn as Bézier curves with arrows; the greater the latency, the higher the arc. The line width reflects the call frequency (a thin line represents a single call, and a thick line represents a high-frequency call). This dual-coordinate mapping mechanism allows the view to simultaneously carry the real timeline and the abstract service architecture. Users can both track the evolution of events along the timeline and observe the distribution of responsibilities along the service layers, achieving multi-dimensional observability.

[0046] In one possible implementation, after forming the chronologically ordered log sequence, the multi-topic log link aggregation method further includes: S1031: For each target log entry in the log sequence, parse its structured fields. If the value of the field belongs to the preset error level enumeration set, then mark the metadata of the target log entry as an error level log.

[0047] Here, for each target log entry in the log sequence, its structured fields are parsed. If the value of the field belongs to the preset error level enumeration set, the metadata of the target log entry is marked as an error level log.

[0048] S1032: If the value of this field does not belong to the preset error level enumeration set, then perform error keyword regular expression matching on the original text content of the target log entry. If the error keyword regular expression matching is satisfied, then mark the metadata of the target log entry as an error level log.

[0049] The error keyword regular expression matching includes display exceptions, status exceptions, and system failures. When any rule matches successfully, the target log entry is marked as an error-level log.

[0050] S1033: The total number of log entries marked as error level logs within a unit time window is counted. When the total number of log entries exceeds a preset alarm threshold, an alarm event is generated. The alarm event includes the name of the service that made the error, high-frequency error keywords, and associated business identifiers.

[0051] Here, the total number of log entries marked as error level within a unit time window is counted. When the total number of log entries exceeds the preset alarm threshold, an alarm event is generated.

[0052] In one possible implementation, after generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: Based on the service name and time interval of adjacent target log entries, the call pattern between two adjacent target log entries is determined, and arrows are marked to indicate the call flow; the time difference between adjacent target log entries is determined, and when the time difference exceeds a preset threshold, a high latency label is marked next to the corresponding service node; when any target log entry is identified as an error-level log, its service node is highlighted, and a red dashed line is added to connect it to its downstream service nodes.

[0053] Here, service heterogeneity and timing reasonableness checks are performed on the service names and time intervals of adjacent target log entries. The service heterogeneity check ensures that multiple logs within the same service do not constitute cross-service calls. For adjacent log pairs that pass the time-space check (l... i ,l i+1 Based on the combination of `service_name` and the range of `delta` values, the system matches the following predefined call patterns: direct call, asynchronous call, round-robin response, and error propagation. For each successfully matched call pattern, the system generates a structured call edge.

[0054] It should be noted that the process of matching the invocation pattern is as follows: i) Direct invocation pattern: when timestamp i+1 With timestamp i When the interval between them is ∈ (0, 500] ms, it is determined as `service_name`. i `for`service_name i+1 Direct invocation of `. ii) Asynchronous callback mode: when timestamp i+1 timestamp i The interval between them ∈ (500, 5000] ms and log l i When a field contains the keywords `async` or `callback`, it is determined to be an asynchronous callback. iii) Polling response mode: When the timestampi+1 timestamp i The interval between them ∈ (1000, 5000] ms and log l i+1 When a field contains the keyword `poll` or `status=success`, it is determined to be a polling response. iv) Error propagation mode: When l i+1 The log level (log_level) belongs to a preset set of error levels, and l i If the raw log field contains the keywords `calling` or `invoke`, it is considered a false propagation.

[0055] In one possible implementation, after generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: responding to a user's operation command on any service node and displaying the original log content and context log fragments of the target log entry corresponding to that service node.

[0056] To improve the accuracy of fault location and debugging efficiency, this invention provides a fine-grained log interaction mechanism for service nodes. This mechanism allows users to perform click / hover / right-click operations on any service node (such as an authentication log in `auth_log`) in a visualized call chain diagram, and responds in real time to display: (a) the complete original log content of the target log entry corresponding to the service node; (b) context log fragments dynamically extracted based on time proximity, centered on the target log, including the N preceding and M following logs of the same topic (i.e., consecutive log records with the closest timestamps within the same log topic table), and optional cross-topic related logs (such as 1-2 logs adjacent to the caller / callee).

[0057] It should be noted that the context log fragment is not statically preset, but rather a dynamically generated structured log set based on the following three constraints: 1. Time proximity: Based on the timestamp `t0` of the target log, search for logs in the same log topic table (such as `auth_log`) that satisfy `|t0`. 1. **Log entries with t0|≤Δt:** where `Δt` defaults to 3 seconds (configurable), ensuring context coverage of typical RPC call round-trip windows. 2. **Topic Consistency:** Context logs must originate from the same topic table as the target log (i.e., both belong to the `auth_log` table) to ensure consistency in log format, field semantics, and service context, avoiding cross-topic semantic mixing. 3. **Business Identifier Continuity:** All context logs must share the same business identifier value (e.g., `trace_id='trace_abc123'`) with the target log to prevent logs from being mixed due to proximity in time but unrelated business logic (e.g., logs from different requests processed by the same service).

[0058] When a user triggers a node operation, the system directly locates and deserializes the `raw_log` column using a unique primary key (such as topic, partition, offset), bypassing the structured parsing process and restoring the original log format (including spaces, newlines, special characters, multi-line stacks, etc.), ensuring that developers can directly reproduce the situation.

[0059] The core benefits of this application lie in breaking through the limitations of single-topic queries in traditional log platforms. By using business identifiers (such as trace_id, order_id) as links, it accurately retrieves, globally sorts by time, and aggregates logs from multiple topic tables (such as gateway_log, auth_log, etc.) to present a complete end-to-end log stream of a single transaction. This significantly improves the observability and fault location efficiency of distributed systems without relying on APM tools or modifying existing applications. Operations personnel can view the call order, the time consumption of each stage, and abnormal nodes with one click. It also has high scalability, strong security, and low deployment threshold, truly realizing a lightweight intelligent operation and maintenance capability of "observable with logs".

[0060] This application provides a method for multi-topic log link aggregation based on business identifiers. The method includes: performing cross-table searches in parallel across multiple pre-built structured log topic tables based on business identifiers to identify multiple target log entries in each structured log topic table that match the business identifiers; merging the multiple target log entries into a result set; sorting the target log entries in the result set globally in ascending order based on a timestamp field to form a log sequence arranged in chronological order; and generating a visual view of the transaction link based on the log sequence. The visual view uses a time axis as the horizontal axis and service nodes as the vertical axis, with each target log entry mapped to a service node on the time axis with a status marker, the status marker including a log level identifier and an error keyword highlighting identifier. This solution addresses the limitation of existing log systems, which can only perform single-topic queries and cannot reconstruct the complete call chain. It automatically associates log entries from multiple log topics using a unified business identifier, sorts them globally by timestamp, and finally presents the complete call chain in a visual manner. This is the first time that a cross-system transaction process can be reconstructed using only log data. It solves the problem that traditional log systems cannot aggregate logs from multiple topics, and enables the automatic aggregation and display of logs scattered across multiple topics in chronological order, based on a single transaction, thereby improving system observability and troubleshooting efficiency.

[0061] Please see Figure 3 , Figure 4 , Figure 3 This is one of the structural schematic diagrams of a multi-topic log link aggregation device based on a service identifier provided in an embodiment of this application; Figure 4 This is a second schematic diagram of a multi-topic log link aggregation device based on service identifiers provided in an embodiment of this application. Figure 3 As shown, the multi-topic log link aggregation device 300 based on business identifiers includes: The cross-table retrieval module 310 is used to perform cross-table retrieval in parallel in multiple pre-built structured log topic tables based on business identifiers, and to determine multiple target log entries in each structured log topic table that match the business identifiers. The global sorting module 320 is used to merge multiple target log entries into a result set, and to sort the target log entries in the result set in global ascending order based on the timestamp field to form a log sequence arranged in chronological order. The view generation module 330 is used to generate a visual view of the transaction chain based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

[0062] Furthermore, the business identifier input by the user is used to perform parallel cross-table searches in multiple pre-built structured log topic tables to determine the target log entries in each structured log topic table that match the business identifier, including: A unified query syntax is constructed, and the business identifier is used as a query condition, which is synchronously distributed to all structured log topic tables; A combined index based on business identifier and timestamp is enabled for each structured log topic table. When the business identifier is detected to be missing in a certain structured log topic table, the system automatically downgrades to a fuzzy matching strategy to determine the log entries in each structured log topic table that match the business identifier.

[0063] Furthermore, such as Figure 4 As shown, the multi-topic log link aggregation device 300 based on business identifiers also includes a construction module 340, which constructs the structured log topic table through the following steps: Configure a unified logging standard for each distributed subsystem; Based on the unified log specification, an independent log topic table is created for each of the distributed subsystems; wherein, the log topic table corresponds to a physically isolated data storage unit, each log topic table adopts a columnar storage structure, and the timestamp field and the business identifier field jointly establish a composite primary key or unique index; During the log collection phase, the log collector performs field standardization on the original logs and writes the standardized log entries into the corresponding log topic table, thus completing the construction of the structured log topic table.

[0064] Furthermore, such as Figure 4 As shown, the multi-topic log link aggregation device 300 based on business identifiers also includes a fault detection module 350, which is used for: For each target log entry in the log sequence, its structured fields are parsed. If the value of the field belongs to a preset error level enumeration set, the metadata of the target log entry is marked as an error level log. If the value of this field does not belong to the preset error level enumeration set, then perform error keyword regular expression matching on the original text content of the target log entry. If it matches the error keyword regular expression matching, then mark the metadata of the target log entry as an error level log. The total number of log entries marked as error level within a unit time window is counted. When the total number of log entries exceeds a preset alarm threshold, an alarm event is generated. The alarm event includes the name of the service that made the error, high-frequency error keywords, and associated business identifiers.

[0065] Furthermore, such as Figure 4 As shown, the multi-topic log link aggregation device 300 based on business identifiers also includes a view display module 360, which is used for: Based on the service name and time interval of adjacent target log entries, the call pattern between two adjacent target log entries is determined, and arrows are marked to indicate the call flow. Determine the time difference between adjacent target log entries. When the time difference exceeds a preset threshold, mark the corresponding service node with a high latency label. When any target log entry is identified as an error-level log, its service node is highlighted, and a red dashed line is added to connect it to its downstream service nodes.

[0066] Furthermore, the view display module 360 ​​is used for: In response to user commands to any service node, display the original log content and context log fragments of the target log entry corresponding to that service node.

[0067] Furthermore, the global sorting module 320 is also used for: Before sorting, all timestamp fields are standardized to convert local timestamps from different distributed subsystems to UTC time and check whether their deviation exceeds the NTP synchronization tolerance threshold. If an out-of-range timestamp exists, the timestamp is marked as a suspicious time point, and a dashed border is displayed in the visualization view to prompt the user to verify the credibility of the target log entry.

[0068] This application provides a multi-topic log link aggregation device based on a business identifier. The multi-topic log link aggregation device includes: a cross-table retrieval module, used to perform cross-table retrieval in parallel on multiple pre-built structured log topic tables based on the business identifier, to determine multiple target log entries in each structured log topic table that match the business identifier; a global sorting module, used to merge the multiple target log entries into a result set, and to perform global ascending sorting on the target log entries in the result set based on a timestamp field, forming a log sequence arranged in chronological order; and a view generation module, used to generate a visual view of the transaction link based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with a status mark on the time axis, the status mark including a log level identifier and an error keyword highlighting identifier. This solution addresses the limitation of existing log systems, which can only perform single-topic queries and cannot reconstruct the complete call chain. It automatically associates log entries from multiple log topics using a unified business identifier, sorts them globally by timestamp, and finally presents the complete call chain in a visual manner. This is the first time that a cross-system transaction process can be reconstructed using only log data. It solves the problem that traditional log systems cannot aggregate logs from multiple topics, and enables the automatic aggregation and display of logs scattered across multiple topics in chronological order, based on a single transaction, thereby improving system observability and troubleshooting efficiency.

[0069] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0070] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, they can perform the operations described above. Figure 1 The steps of the multi-topic log link aggregation method based on business identifiers in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0071] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the multi-topic log link aggregation method based on business identifiers in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0072] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0074] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0076] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for multi-topic log link aggregation based on business identifiers, characterized in that, The multi-topic log link aggregation method includes: Based on the business identifier, cross-table retrieval is performed in parallel in multiple pre-built structured log topic tables to determine multiple target log entries in each structured log topic table that match the business identifier. Multiple target log entries are merged into a result set, and the target log entries in the result set are sorted globally in ascending order based on the timestamp field to form a log sequence arranged in chronological order. A visual view of the transaction chain is generated based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

2. The multi-topic log link aggregation method according to claim 1, characterized in that, The user-input-based business identifier performs parallel cross-table searches across multiple pre-built structured log topic tables to determine target log entries in each structured log topic table that match the business identifier, including: A unified query syntax is constructed, and the business identifier is used as a query condition, which is synchronously distributed to all structured log topic tables; A combined index based on business identifier and timestamp is enabled for each structured log topic table. When the business identifier is detected to be missing in a certain structured log topic table, the system automatically downgrades to a fuzzy matching strategy to determine the log entries in each structured log topic table that match the business identifier.

3. The multi-topic log link aggregation method according to claim 1, characterized in that, The structured log topic table is constructed using the following steps: Configure a unified logging standard for each distributed subsystem; Based on the unified log specification, an independent log topic table is created for each of the distributed subsystems; wherein, the log topic table corresponds to a physically isolated data storage unit, each log topic table adopts a columnar storage structure, and the timestamp field and the business identifier field jointly establish a composite primary key or unique index; During the log collection phase, the log collector performs field standardization on the original logs and writes the standardized log entries into the corresponding log topic table, thus completing the construction of the structured log topic table.

4. The multi-topic log link aggregation method according to claim 1, characterized in that, After forming the log sequence arranged in chronological order, the multi-topic log link aggregation method further includes: For each target log entry in the log sequence, its structured fields are parsed. If the value of the field belongs to a preset error level enumeration set, the metadata of the target log entry is marked as an error level log. If the value of this field does not belong to the preset error level enumeration set, then perform error keyword regular expression matching on the original text content of the target log entry. If it matches the error keyword regular expression matching, then mark the metadata of the target log entry as an error level log. The total number of log entries marked as error level within a unit time window is counted. When the total number of log entries exceeds a preset alarm threshold, an alarm event is generated. The alarm event includes the name of the service that made the error, high-frequency error keywords, and associated business identifiers.

5. The multi-topic log link aggregation method according to claim 1, characterized in that, After generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: Based on the service name and time interval of adjacent target log entries, the call pattern between two adjacent target log entries is determined, and arrows are marked to indicate the call flow. Determine the time difference between adjacent target log entries. When the time difference exceeds a preset threshold, mark the corresponding service node with a high latency label. When any target log entry is identified as an error-level log, its service node is highlighted, and a red dashed line is added to connect it to its downstream service nodes.

6. The multi-topic log link aggregation method according to claim 1, characterized in that, After generating a visual view of the transaction chain based on the log sequence, the multi-topic log chain aggregation method further includes: In response to user commands to any service node, display the original log content and context log fragments of the target log entry corresponding to that service node.

7. The multi-topic log link aggregation method according to claim 1, characterized in that, The global ascending sort also includes: Before sorting, all timestamp fields are standardized to convert local timestamps from different distributed subsystems to UTC time and check whether their deviation exceeds the NTP synchronization tolerance threshold. If an out-of-tolerance timestamp exists, the timestamp is marked as a suspicious time point, and a dashed border is displayed in the visualization view to prompt the user to verify the credibility of the target log entry.

8. A multi-topic log link aggregation device based on business identifiers, characterized in that, The multi-topic log link aggregation device includes: The cross-table retrieval module is used to perform cross-table retrieval in parallel in multiple pre-built structured log topic tables based on business identifiers, and to determine multiple target log entries in each structured log topic table that match the business identifiers. The global sorting module is used to merge multiple target log entries into a result set, and to sort the target log entries in the result set in global ascending order based on the timestamp field to form a log sequence arranged in chronological order. The view generation module is used to generate a visual view of the transaction chain based on the log sequence; wherein, the visual view uses the time axis as the horizontal axis and the service node as the vertical axis, and each target log entry is mapped to a service node with status markers on the time axis, the status markers including log level identifiers and error keyword highlighting identifiers.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the multi-topic log link aggregation method based on service identifiers as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multi-topic log link aggregation method based on any one of claims 1 to 7.

Images