A multi-source call chain-oriented application-database-cache topology graph construction method

Abstract

This invention discloses a method for constructing an application-database-cached topology graph for multi-source call chains. This invention incorporates multiple source trace nodes, such as HTTP call nodes, application service nodes, SQL access nodes, Redis access nodes, and RPC call nodes, into a unified graph construction process. It achieves stable single-node rendering of the same resource in the topology graph through differentiated node identity normalization rules, merges repeated calls to express call intensity through edge merging rules based on start point, end point, and edge type, automatically labels abnormal node states by aggregating node errors and associated edge errors, and achieves bidirectional backtracking of graph display and original call details by retaining the original trace node identifier set in nodes and edges.

Description

Technical Field

[0001] This invention belongs to the field of distributed system monitoring, specifically involving a method for constructing an application-database-cached topology graph for multi-source call chains. Background Technology

[0002] A complete call chain typically contains multiple types of nodes, including client requests, application services, database access, cache access, and remote calls. Without unified modeling, the front end often only sees discrete events, making it difficult to quickly identify the true link structure. Existing topology construction schemes often combine "node display" and "node identity normalization," resulting in the same database, remote service, or application being rendered as multiple nodes, leading to significant graph noise. Multiple SQL, Dubbo, or Redis calls without edge merging can easily generate numerous duplicate edges in the graph, affecting readability and making it difficult to stably represent call intensity. Existing technologies typically lack a unified graph construction mechanism that includes "multi-source trace nodes -> node identity normalization -> same-edge merging and counting -> node anomaly labeling -> graph object output."

[0003] Inconsistent node identity rules lead to duplicate graph representations of the same resource. A lack of a unified convergence path for multi-source events results in inconsistencies between the graph and detailed data. Edge relationships are not consistently merged based on start point, end point, and event type, making it difficult to accurately represent call intensity. Node anomaly information is not uniformly mapped to the graph display status, leading to low troubleshooting efficiency. Summary of the Invention

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: S1. Obtain the set of trace nodes used for graph construction.

[0005] Obtain a set of trace nodes for graph construction, including at least application call nodes, database access nodes, cache access nodes, and remote call nodes. Each trace node carries a traceId (call chain identifier), a traceNodeId (identifier of a single node within the trace), and information on the node's event type, label field, and call semantics. The trace nodes can represent HTTP request events, application method call events, SQL access events, Redis access events, and RPC call events.

[0006] S2. Identify the node type of each Trace node, and generate the graph node nodeId according to the corresponding resource identity normalization rule based on the node type.

[0007] The system identifies the node type of each trace node based on the event type, tag field, and call semantics of the original trace node. Node types include CLIENT (client node), APP (application node), DB (database node), REDIS (cached node), and REMOTE (remote service node). The node type identification rules are as follows:

[0008] After identifying the node type, the corresponding resource identity normalization rule is selected according to the node type to generate a unique nodeId for each graph node. Different node types use different nodeId generation rules: Application node: nodeId = "APP:" + appId Database node: nodeId = "DB:" + addressIp + ":" + port + ":" + dbName Cache node: nodeId = "REDIS:" + host + ":" + port Remote service node: nodeId = "REMOTE:" + interfaceName or serviceName Client node: nodeId = "CLIENT:" + normalize(userAgent or clientIp) The following is an example of node identity normalization:

[0009] Through the above normalization process, the same database, the same cache instance, the same application, or the same remote service are rendered as only one stable graph node in multiple Trace events, effectively reducing duplicate graph rendering.

[0010] S3. Merge trace nodes with the same nodeId into the same graph node and accumulate statistical information.

[0011] For trace nodes with the same nodeId, they are merged into a single graph node, and the associated sourceTraceNodeIds, call count, error count (selfErrorCount), and totalDuration are accumulated. The merging logic is as follows: If nodeId already exists in the graph, then: graphNode.count += 1; graphNode.selfErrorCount += currentError ? 1 : 0; graphNode.totalDuration += currentDuration; graphNode.sourceTraceNodeIds += traceNodeId; If nodeId does not exist in the graph, a new graph node is created.

[0012] The data structure of the graph node object GraphNode is as follows: GraphNode = { nodeId, / / Unique identifier for graph node nodeType, / / Node type displayName, / / Display Name count, / / Number of times it is called selfErrorCount, / / The number of errors made by the user itself totalDuration, / / Total Duration avgDuration, / / Average Duration status, / / Node status: NORMAL / WARN / ERROR sourceTraceNodeIds / / The set of associated original trace node identifiers } S4. Generate graph edges based on the parent-child relationship or call relationship between Trace nodes, and form edgeKey with sourceNodeId, targetNodeId and edgeType.

[0013] Graph edges are generated based on parent-child relationships or call relationships between trace nodes. Edge types include:

[0014] The edge identity (i.e., the edge merge key edgeKey) is determined by the source nodeId, the destination nodeId, and the edge type edgeType: edgeKey = sourceNodeId + "->" + targetNodeId + ":" + edgeType.

[0015] S5. Merge graph edges with the same edgeKey, and accumulate count, errorCount, and totalDuration.

[0016] Edges with the same edgeKey are merged. The cumulative call count (count), error count (errorCount), and total duration (totalDuration) represent the number of duplicate and exception calls, respectively. The edge merging rules are as follows: If edgeKey already exists, then: edge.count += 1; edge.totalDuration += currentDuration; edge.errorCount += currentError ? 1 : 0; edge.sourceTraceNodeIds += traceNodeId; If edgeKey does not exist, a new graph edge will be created.

[0017] The data structure of the graph edge object GraphEdge is as follows: GraphEdge = { edgeKey, / / Edge merge key sourceNodeId, / / Identifier of the starting node targetNodeId, / / Endpoint node identifier edgeType, / / Edge type count, / / Number of times it is called errorCount, / / Number of exceptions totalDuration, / / Total Duration avgDuration, / / Average Duration sourceTraceNodeIds / / The set of associated original trace node identifiers } By using the edge merging rules described above, multiple identical calls are expressed by adding a count to an edge to represent the call intensity, thus avoiding a large number of duplicate lines in the graph.

[0018] S6. Perform status marking on the graph node based on its own error information and the errorCount of its associated edges.

[0019] The node state is determined by the number of errors of the node itself, the number of errors of associated edges, the average execution time, and the number of calls. The state determination rules are as follows: If node.selfErrorCount > 0 or relatedEdge.errorCount > 0, then node.status = ERROR; Otherwise, if avgDuration > durationThresholdMs or count > countThreshold, then node.status = WARN; Otherwise, node.status = NORMAL.

[0020] The state determination table is as follows:

[0021] Through the aforementioned state markers, the topology graph not only displays topological relationships but also expresses anomaly aggregation and fault propagation direction.

[0022] S7. Outputs a topology graph object containing nodes, edges, and sourceTraceNodeIds.

[0023] The output is a TopologyGraph object containing nodes, edges, and sourceTraceNodeIds, allowing users to navigate between the graph display and the original Trace details. The data structure of the TopologyGraph object is as follows: TopologyGraph = { traceId, / / Call chain identifier nodes: GraphNode[], / / List of graph nodes edges: GraphEdge[], / / List of graph edges defaultNode, / / Default node (entry node) graphSummary / / Graph Summary Statistics } The graphSummary statistics include the following fields:

[0024] Both graph nodes and edges retain sourceTraceNodeIds (the set of associated original trace node identifiers). When a user clicks on a topology node or edge, the system uses sourceTraceNodeIds to jump back to the original trace node, SQL content, Redis command, or remote call details, enabling bidirectional backtracking between graph display and original call details.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Stable and uniform node representation, resulting in a concise and clear graph. The same database instance, Redis instance, or application service, regardless of how many times it appears in the trace, is rendered as only one node in the topology graph. Through differentiated nodeId generation rules, stable normalization is achieved for application nodes based on appId, database nodes based on address, port, and database name, cache nodes based on host and port, and remote service nodes based on interface name. This significantly reduces redundant nodes in the graph, making the topology structure clearer. 2. Edge-level merging to express call intensity. Multiple SQL calls to the same database, multiple accesses to the same Redis instance, or multiple RPC calls to the same remote service from the same application are merged into a single graph edge. Each edge carries a count and an error count, allowing users to intuitively see the call frequency and error rate. The edgeKey mechanism, composed of the sourceNodeId, targetNodeId, and edgeType triples, ensures stable merging of edge relationships, avoiding a large number of duplicate lines in the graph. 3. Automatic aggregation and marking of abnormal states. The state of a graph node is determined by both the node's own errors and errors in its associated edges. When a node has an error or its associated edges are abnormal, the node is automatically marked as ERROR and highlighted. Users can quickly locate faulty nodes without having to expand the details of each span, significantly improving troubleshooting efficiency. 4. Supports bidirectional backtracking of original evidence. Both graph nodes and edges store a list of sourceTraceNodeIds, recording the identifier of the associated original trace node. When a user clicks on a topology node or edge, the system can directly jump to the corresponding SQL statement, Redis command, or RPC message details based on the sourceTraceNodeIds, achieving seamless linkage between graph display and original call details.

[0026] 5. Unified graph construction process and consistent handling of multiple source events. Events such as HTTP requests, SQL accesses, Redis accesses, and RPC calls are no longer displayed separately, but are incorporated into the same graph construction process and uniformly output as a single TopologyGraph object. The front end can complete the display of the entire topology in a single rendering, ensuring the consistency between the graph and the detailed data. Attached Figure Description

[0027] Figure 1 This is an overall flowchart of the application-database-cacher topology graph construction method for multi-source call chains according to the present invention; Figure 2 This is a flowchart of the Trace node type identification and nodeId generation process in this invention; Figure 3 This is a schematic diagram of the node merging process in this invention; Figure 4 This is a schematic diagram of the graph edge merging process in this invention; Figure 5 This is a schematic diagram of the topology graph object structure output by the present invention. Detailed Implementation

[0028] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Example

[0029] Implementation Method 1: System Architecture and Data Flow This invention can be deployed in the backend services of an application performance monitoring (APM) system. The system receives Span data reported from distributed tracing components (such as Jaeger, Zipkin, or a self-developed tracing system) and extracts Trace node information from the Span data. Each Trace node contains at least traceId, traceNodeId, event type, call start and end time, status code (success / failure), call parameters (such as SQL statement, Redis command, RPC interface name), and associated resource identifiers (such as appId, databaseName, host, port, etc.).

[0030] The system follows Figure 1 The process shown executes node type identification, nodeId generation, graph node merging, graph edge generation and merging, node status marking, and topology graph object output in sequence.

[0031] Implementation Method 2: Specific Implementation of Node Type Identification The node type identification module identifies the node based on its event type label and extended fields. Specifically: For HTTP type Spans, if their parent Span is empty or is an entry type, they are identified as CLIENT type; for Spans carrying appId or serviceName tags, they are identified as APP type; for Spans carrying SQL statements or databaseName tags, they are identified as DB type; for Spans carrying Redis commands or host tags, they are identified as REDIS type; and for Spans carrying Dubbo, Feign, or RPC interface name tags, they are identified as REMOTE type.

[0032] Implementation Method 3: Specific Implementation of nodeId Generation The nodeId generation module selects the corresponding generation rule based on the node type. For application nodes, the system extracts the appId or serviceName from the Span tag and generates a nodeId in the format "APP:" + appId; for database nodes, the system extracts the addressIp, port, and dbName and generates a nodeId in the format "DB:" + addressIp + ":" + port + ":" + dbName; for cache nodes, the system extracts the host and port and generates a nodeId in the format "REDIS:" + host + ":" + port; for remote service nodes, the system extracts the interfaceName or serviceName and generates a nodeId in the format "REMOTE:" + interfaceName; for client nodes, the system extracts the User-Agent or clientIp and normalizes it (e.g., normalizing "Chrome / 120.0.0.0" to "chrome") and generates a nodeId in the format "CLIENT:" + normalize (identifier).

[0033] Implementation Method 4: Specific Implementation of Node Merging and Statistics The graph node merging module maintains a hash table with nodeId as the key. For each input trace node, its nodeId is calculated and searched in the hash table. If found, the existing graph node's count (incremented by 1), selfErrorCount (incremented by 1 if the current trace node has an error), totalDuration (accumulated current time), and sourceTraceNodeIds (added with the current traceNodeId) are updated. If not found, a new graph node is created, initializing count to 1, selfErrorCount to the current error status, totalDuration to the current time, and sourceTraceNodeIds to contain the current traceNodeId. After processing all trace nodes, the avgDuration of each graph node is calculated as totalDuration / count.

[0034] Implementation Method 5: Specific Implementation of Edge Generation and Merging The graph edge generation module iterates through the parent-child relationships between trace nodes. For each pair of trace nodes with a parent-child call relationship, it obtains the parent node's nodeId as the sourceNodeId and the child node's nodeId as the targetNodeId, and determines the edgeType (HTTP_CALL, SQL_CALL, REDISS_CALL, or RPC_CALL) based on the call type. It calculates edgeKey = sourceNodeId + "->" + targetNodeId + ":" + edgeType. The graph edge merging module maintains a hash table with edgeKey as the key. For each generated edge, it searches the hash table; if found, it updates the count, errorCount, totalDuration, and sourceTraceNodeIds of the existing graph edge; otherwise, it creates a new graph edge.

[0035] Implementation Method Six: Specific Implementation of Node State Marking After merging nodes and edges, the node status marking module iterates through all graph nodes, checks if the `selfErrorCount` of each node is greater than 0, and retrieves the sum of the `errorCount`s of all graph edges originating from or ending at that node. If `selfErrorCount` > 0 or the sum of the `errorCount`s of associated edges > 0, the node is marked as ERROR; otherwise, if `avgDuration` exceeds a preset time threshold (e.g., 1000ms) or `count` exceeds a preset call count threshold (e.g., 1000 times), it is marked as WARN; otherwise, it is marked as NORMAL. The time threshold and call count threshold can be configured by the system administrator according to the actual business scenario.

[0036] Implementation Method Seven: Specific Implementation of Evidence Retrospection When outputting the TopologyGraph object, the topology graph object output module ensures that the sourceTraceNodeIds field of each graph node and edge is fully preserved. When the front-end displays the topology graph, each node and edge is associated with a list of sourceTraceNodeIds. When a user clicks on a node or edge, the front-end sends a query request carrying the sourceTraceNodeIds to the back-end. The back-end retrieves the corresponding Span details (including SQL statements, Redis commands, RPC messages, exception stacks, etc.) from the original Trace storage based on these traceNodeIds and returns them to the front-end for display, enabling fast drill-down from the topology graph to the original call details.

[0037] Implementation Method 8: Multiple Replacement Options In the node type identification process, in addition to identification based on event type labels, identification can also be based on the component type field of the Span. For example, a component with the name "gRPC" is identified as the REMOTE type, and a component with the name "jedis" is identified as the REDIS type.

[0038] In the node status marking process, in addition to using a fixed threshold, a dynamic baseline threshold can also be used, that is, dynamically calculating the time consumption threshold and call count threshold of the current node based on historical statistical data to achieve adaptive alarms.

[0039] In terms of edge type definition, in addition to HTTP_CALL, SQL_CALL, REDIS_CALL and RPC_CALL, other edge types can be extended according to the actual system architecture, such as MQ_CALL (message queue call) and GRPC_CALL (gRPC call).

[0040] The specific embodiments described above are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should understand that various modifications, equivalent substitutions, and improvements can be made to the invention without departing from its principles, and all such modifications, substitutions, and improvements should fall within the scope of protection defined by the claims.

Claims

1. A method for constructing an application-database-cacher topology graph for multi-source call chains, characterized in that, include: S1. Obtain a set of Trace nodes for graph construction, wherein the Trace nodes include at least application call nodes, database access nodes, cache access nodes, and remote call nodes; S2. Identify the node type of each Trace node, select the corresponding resource identity normalization rule according to the node type, and generate a unique identifier nodeId for the graph node. Database nodes generate nodeId based on address, port, and database name; cache nodes generate nodeId based on host and port; and application nodes generate nodeId based on appId. S3. Merge trace nodes with the same nodeId into the same graph node, and accumulate the sourceTraceNodeIds, call count, error count selfErrorCount, and totalDuration associated with the graph node. S4. Generate graph edges based on the parent-child relationship or call relationship between Trace nodes, and use sourceNodeId, targetNodeId and edgeType to form the edge merging key edgeKey; S5. Merge graph edges with the same edgeKey. The cumulative number of calls count, errorCount, and totalDuration are used to represent the number of repeated calls and the number of abnormal calls. S6. Based on the error information of the graph node itself and the errorCount of its associated edges, perform status marking on the graph node to generate NORMAL, WARN or ERROR status. S7. Outputs a topology graph object containing nodes, edges, and sourceTraceNodeIds, enabling the graph display and the original Trace details to jump between each other.

2. The method according to claim 1, characterized in that, In step S2, the nodeId of the application node is generated by concatenating "APP:" and appId.

3. The method according to claim 1, characterized in that, In step S2, the nodeId of the database node is generated by concatenating "DB:", addressIp, port, and database namedbName.

4. The method according to claim 1, characterized in that, In step S2, the nodeId of the cache node is generated by concatenating "REDIS:", host, and port.

5. The method according to claim 1, characterized in that, In step S4, edgeType includes at least HTTP_CALL, SQL_CALL, REDISS_CALL, and RPC_CALL.

6. The method according to claim 1, characterized in that, In step S6, the conditions for determining that a node is in an ERROR state include that the node's own error count (selfErrorCount) is greater than zero or the sum of the error counts (errorCount) of its associated edges is greater than zero.

7. The method according to claim 1, characterized in that, In step S7, both graph nodes and graph edges store sourceTraceNodeIds, which are used to trace back to the original Trace node.

8. The method according to claim 1, characterized in that, In step S2, the nodeId of the remote service node is generated by concatenating "REMOTE:" with the interface name interfaceName or the service name serviceName.

9. The method according to claim 1, characterized in that, In step S2, the nodeId of the client node is generated by concatenating "CLIENT:" with the normalized User-Agent or the client IP.

10. The method according to claim 1, characterized in that, In step S7, the topology graph object also includes graphSummary graph summary statistics, which includes the number of application nodes (appNodeCount), the number of database nodes (dbNodeCount), the number of cache nodes (redisNodeCount), the number of merged edges (edgeCount), the number of abnormal nodes (errorNodeCount), and the number of abnormal edges (errorEdgeCount).

Images