Method for automatically generating application call topology based on APM link data

Abstract

The invention relates to the related technical field of application performance monitoring, in particular to a method for automatically generating an application call topology based on APM link data, which comprises the following steps of: acquiring link tracking data of an APM system, preprocessing the data, identifying and modeling call nodes, constructing call edges, optimizing topological layout, performing visual rendering and the like. And automatic topology generation and dynamic display of the application calling relationship of the distributed system are realized. According to the method, the problems of low efficiency and poor real-time performance of traditional manual topology combing are solved, the application calling dependency and performance indexes can be accurately and efficiently presented, and a visual topology support is provided for distributed system monitoring, fault positioning and performance optimization.

Description

Technical Field

[0001] This invention relates to the field of application performance monitoring technology, and in particular to a method for automatically generating application call topology based on APM link data. Background Technology

[0002] In a distributed system architecture, applications typically consist of multiple microservices or modules, which interact with each other via complex calls over a network. Understanding the application call topology is a crucial prerequisite for achieving observability, fault location, and performance optimization in distributed systems.

[0003] Traditional application topology mapping relies on manual analysis, where technical personnel manually draw topologies based on system design documents, code logic, or experience. However, this approach has significant drawbacks: firstly, distributed systems are highly dynamic (e.g., service instance scaling up or down, changes in call relationships), and manually maintained topologies cannot reflect the actual system status in real time; secondly, manual mapping is inefficient and prone to inaccuracies due to subjective judgment or information lag, a problem that is even more pronounced in large-scale distributed systems.

[0004] With the development of APM technology, systems can collect rich tracing data (such as request call chains, service call sequences, response times, call frequencies, etc.). How to efficiently and accurately generate application call topologies based on this automatically collected tracing data has become a pressing technical problem. While some existing methods attempt to generate topologies using tracing data, they still fall short in areas such as call relationship integrity identification, topology layout rationality and readability, and dynamic topology update mechanisms, making it difficult to meet the requirements of complex distributed systems for topology visualization. Summary of the Invention

[0005] The purpose of this invention is to provide a method for automatically generating application call topology based on APM link data, so as to solve the problems of low efficiency and poor real-time performance of manual application call topology sorting in the prior art, as well as the problems of incomplete call relationship identification and unreasonable topology layout in the existing automatic generation methods.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for automatically generating application call topology based on APM link data, comprising the following steps:

[0007] Step S1: Collect link tracing data of each service in the distributed system in real time through probes or log collection components deployed in the application service. The link tracing data includes at least the unique request identifier, the identifier of the service initiating the call, the identifier of the target service, the instance information of the service initiating the call, the instance information of the target service, the call timestamp, the call time, and the call result.

[0008] Step S2: Clean and preprocess the collected raw link data, including data cleaning, data deduplication, and format standardization;

[0009] Step S3: Based on the preprocessed link data, identify the application service nodes in the distributed system and model each node.

[0010] Step S4: Construct call edges in the topology based on the call relationships between services in the link data;

[0011] Step S5: Use a graph layout algorithm to lay out the identified nodes and edges, and generate a visualized topology structure by selecting and optimizing the layout algorithm.

[0012] Step S6: Visualize and render the layout topology, and establish a dynamic update mechanism with timed updates and event-driven updates.

[0013] Preferably, in step S2:

[0014] Data cleaning includes: filtering invalid data;

[0015] Data deduplication includes: removing duplicate link records caused by repeated reporting by the collection component based on Trace ID and call sequence;

[0016] Format standardization includes converting link data from different sources into a unified structured data format to facilitate subsequent processing.

[0017] Preferred,

[0018] The specific steps in step S3 of identifying application service nodes in the distributed system include:

[0019] Extract all relevant service names from the Source Service and Target Service fields of the link data, and use them as service nodes in the topology;

[0020] By combining service identifiers and service instance information, each service is associated with its corresponding instance node, thus clarifying the hierarchical relationship between "service and instance".

[0021] The modeling of each node in step S3 includes:

[0022] Define attributes for each node, including at least: unique node identifier, service name, instance information, current alarm status, service type, and historical call frequency statistics.

[0023] Preferably, the step S4 of calling edge construction includes:

[0024] Edge direction determination: If there are call records from service A to service B in the link data, then construct a directed edge from service A to service B;

[0025] Edge attribute assignment: Define attributes for each edge, including at least: call frequency, average call time, call success rate, and most recent call timestamp;

[0026] Edge aggregation and optimization: For multiple call edges in the same direction and with the same source-target, aggregation processing is performed. Attributes such as call frequency and time consumption are statistically aggregated to reduce the number of edges and improve topology readability.

[0027] Preferred,

[0028] The layout algorithm selection in step S5 includes:

[0029] Based on the force-directed layout algorithm, it simulates the "attraction" and "repulsion" between nodes, so that the topology presents a natural and clear distribution. At the same time, combined with the concept of hierarchical layout, nodes with clear calling levels are arranged in layers to enhance the hierarchical readability of the topology.

[0030] The layout optimization strategy in step S5 includes:

[0031] Node clustering: Clustering and displaying multiple instance nodes belonging to the same service to avoid topology clutter caused by too many instances;

[0032] Edge crossing optimization: By adjusting node positions or using edge routing algorithms, edge crossings are reduced, improving topology clarity;

[0033] Highlighting key nodes: Nodes with alarm status or abnormal call frequency / time consumption are highlighted by increasing their size or using a brighter color.

[0034] Preferred,

[0035] The visualization rendering in step S6 includes:

[0036] Node rendering: Different shapes and colors are used to distinguish service types, and light and dark colors or flashing effects are used to indicate alarm status;

[0037] Edge rendering: Different thicknesses are used to represent call frequency, different colors represent average time consumption, and key data on call frequency and average time consumption are marked on the edge;

[0038] Enhanced interactivity: Supports interactive operations such as zooming and panning the topology, highlighting selected nodes, and viewing call details by selecting edges;

[0039] The dynamic update mechanisms for timed updates and event-driven updates in step S6 include:

[0040] Scheduled Updates: Process link data at preset time intervals, repeating steps 2-5 to update the nodes, edges, and attributes of the topology;

[0041] Event-driven updates: When the APM system detects changes in service instances, call relationships, or alarm status, it triggers immediate updates to ensure that the topology reflects the system status in real time.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] 1. High degree of automation: The entire topology generation is completed automatically based on APM link data, without the need for manual intervention, which significantly improves the efficiency of topology construction;

[0044] 2. Strong real-time performance: By combining scheduled updates with event-driven updates, it can reflect changes in service call relationships and status in a distributed system in real time;

[0045] 3. High accuracy: Node identification and edge construction are based on real link tracing data, avoiding subjective errors caused by manual sorting and ensuring that the topology is consistent with the actual call relationship;

[0046] 4. Excellent readability: Through layout algorithm optimization and visualization enhancement, the complex distributed call topology is made clear and easy to read, making it easy for technical personnel to quickly grasp the system call dependencies;

[0047] 5. Strong auxiliary function: The topology integrates performance and status information such as call frequency, time consumption, and alarms, providing intuitive visualization support for fault location and performance optimization of distributed systems. Attached Figure Description

[0048] Figure 1 An example diagram of the application call topology generated for an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Please see Figure 1 This invention provides a technical solution: a method for automatically generating application call topology based on APM link data, comprising the following steps:

[0051] Step S1, APM link data acquisition

[0052] By deploying probes (Agents) or log collection components in application services, link tracing data of each service in the distributed system is collected in real time. The link tracing data includes at least: unique request identifier (Trace ID), source service identifier, target service identifier, source service instance information (such as IP address and port), target service instance information, call timestamp, call time, and call result (success / failure).

[0053] Step S2, Link Data Preprocessing

[0054] The collected raw link data is cleaned and preprocessed, including:

[0055] Data cleaning: Filtering invalid data (such as null values, incorrectly formatted data, and data collected repeatedly);

[0056] Data deduplication: Based on Trace ID and call sequence, duplicate link records caused by repeated reporting by the collection component are removed;

[0057] Format standardization: Convert link data from different sources (such as probes in different languages ​​and log formats) into a unified structured data format to facilitate subsequent processing.

[0058] Step S3, invoke node recognition and modeling

[0059] Based on the preprocessed link data, the application service nodes in the distributed system are identified, and each node is modeled:

[0060] Service node extraction: Extract all relevant service names from the Source Service and Target Service fields of the link data, and use them as service nodes in the topology;

[0061] Service instance association: By combining the service identifier with service instance information (such as IP address and port), each service is associated with its corresponding instance node, clarifying the hierarchical relationship between "service and instance";

[0062] Node attribute modeling: Define attributes for each node (including service nodes and instance nodes), including at least: unique node identifier, service name, instance information (such as IP, port), current alarm status (alarm data association based on APM), service type (such as web service, database service, etc.), historical call frequency statistics, etc.

[0063] Step S4, call edge construction

[0064] Based on the call relationships between services in the link data, construct the call edges in the topology:

[0065] Edge direction determination: If there is a call record from service A to service B in the link data, then construct a directed edge from service A (or its instance) to service B (or its instance);

[0066] Edge attribute assignment: Define attributes for each edge, including at least: call frequency (number of calls per unit time), average call time, call success rate (statistics based on call results), and most recent call timestamp, etc.

[0067] Edge aggregation and optimization: For multiple call edges in the same direction and with the same source-target (such as calls between the same service corresponding to different Trace IDs), aggregation is performed. Attributes such as call frequency and time consumption are statistically aggregated (such as summation and averaging) to reduce the number of edges and improve topology readability.

[0068] Step S5, Topology Layout Algorithm Design and Optimization

[0069] A graph layout algorithm is used to lay out the identified nodes and edges, generating a visual topology structure:

[0070] Layout algorithm selection: Force-directed layout is adopted as the basis to simulate the "attraction" (representing call dependency) and "repulsion" (representing node spacing) between nodes, so that the topology structure presents a natural and clear distribution; at the same time, combined with the concept of hierarchical layout, nodes with clear call hierarchy (such as front-end service → back-end service → database service) are arranged in layers to enhance the readability of the topology hierarchy.

[0071] Layout optimization strategy:

[0072] Node clustering: Clustering and displaying multiple instance nodes belonging to the same service to avoid topology clutter caused by too many instances;

[0073] Edge crossing optimization: By adjusting node positions or using edge routing algorithms (such as orthogonal routing and Bézier curve routing), edge crossings are reduced, and topology clarity is improved;

[0074] Highlighting key nodes: Nodes with alarm status or abnormal call frequency / time consumption are highlighted by increasing their size or using a brighter color.

[0075] Step S6, Topology Visualization and Dynamic Update

[0076] The laid-out topology is then visualized and rendered, and a dynamic update mechanism is established:

[0077] Visual rendering:

[0078] Node rendering: Different shapes and colors are used to distinguish service types (e.g., circles represent web services and rectangles represent database services), and color depth or flashing effects are used to indicate alarm status (e.g., red represents host alarms and orange represents application alarms).

[0079] Edge rendering: Different thicknesses are used to represent call frequency (the thicker the thickness, the more frequent the call), and different colors represent average time consumption (e.g., red indicates excessive time consumption, green indicates normal time consumption), and key data on call frequency and average time consumption are marked on the edge;

[0080] Enhanced interactivity: Supports topology zooming, panning, node selection highlighting (displaying detailed attributes of the node and detailed data of associated edges), edge selection to view call details, and other interactive operations;

[0081] Dynamic update mechanism:

[0082] Scheduled Updates: Process link data at preset time intervals (e.g., 1 minute, 5 minutes), repeating steps 2-5 to update the nodes, edges, and attributes of the topology;

[0083] Event-driven updates: When the APM system detects changes in service instances (such as addition or removal), changes in call relationships, or changes in alarm status, it triggers immediate updates to ensure that the topology reflects the system status in real time.

[0084] Example

[0085] Take the distributed microservice system of an e-commerce platform as an example:

[0086] Step 1: APM Link Data Acquisition

[0087] APM probes are deployed in various microservices of the e-commerce platform (such as user service, order service, product service, payment service, database service, etc.). The probes collect call chain data between services in real time. For example, when a user initiates a "create order" request, the probe records the Trace ID of the request, as well as data such as the Source Service, Target Service, instance IP, call time, and call result of a series of calls from user service → order service → product service → database service, and then from order service → payment service, and reports this data to the APM server.

[0088] Step 2: Link Data Preprocessing

[0089] The APM server preprocesses the collected raw link data:

[0090] Data cleaning: Filter out empty Trace ID records caused by probe anomalies, as well as log entries with incorrect formats;

[0091] Data deduplication: Based on Trace ID and call timestamp, identify and remove duplicate reported data from the same call chain;

[0092] Format standardization: Data from different sources, such as Java probes and Python probes, is converted into structured data using JSON as the unified format. An example is shown below:

[0093] JSON{ "traceId": "123e4567-e89b-12d3-a456-426614174000", "sourceService": "order-service", "sourceInstance": "192.168.1.10:8080", "targetService": "product-service", "targetInstance": "192.168.1.11:9090", "callTimestamp": "2025-09-19T10:30:00.123Z", "callDuration": 50, / / Unit: milliseconds "callResult": "success"}

[0094] Step 3: Invoke node recognition and modeling

[0095] Based on the standardized link data, service nodes and instance nodes are extracted and modeled:

[0096] Service node extraction: Extract service names such as order-service, product-service, user-service, payment-service, and db-service from the sourceService and targetService fields of all link data, and use them as service nodes in the topology;

[0097] Service instance association: Combine the sourceInstance and targetInstance fields to associate each service with its instance. For example, order-service is associated with instances 192.168.1.10:8080 and 192.168.1.12:8080 (multi-instance deployment).

[0098] Node attribute modeling: Define attributes for each node. Taking the order-service instance 192.168.1.10:8080 as an example, its attributes include:

[0099] Node unique identifier: order-service-192.168.1.10:8080;

[0100] Service name: order-service;

[0101] Instance information: IP: 192.168.1.10, Port: 8080;

[0102] Alarm status: None (If APM detects that the CPU utilization of this instance is too high, it will mark it as "host alarm");

[0103] Service type: Business service;

[0104] Historical call frequency: Called 1200 times in the past 5 minutes.

[0105] Step 4: Call edge construction

[0106] Based on the call relationships in the link data, construct directed edges and assign values:

[0107] Determining the direction of the edge: For records in the link data where sourceService is user-service and targetService is order-service, construct a directed edge from user-service to order-service;

[0108] Edge attribute assignment: If the number of calls from user-service to order-service within a unit of time (e.g., 1 minute) is 300, the average time is 60ms, and the success rate is 99.5%, then assign the following values ​​to this edge: call frequency = 300 times / minute, average time = 60ms, success rate = 99.5%;

[0109] Edge aggregation and optimization: If there are multiple user-service→order-service links (corresponding to requests with different Trace IDs), the call frequency of these links is summed and the time consumption is averaged to aggregate them into a single edge, thus avoiding an excessive number of edges.

[0110] Step 5: Topology Layout Algorithm Design and Optimization

[0111] Topology layout is performed using a force-directed layout algorithm combined with a hierarchical layout:

[0112] Force-oriented layout: Each node is regarded as a particle with "mass", and the edges are regarded as "springs" between particles. By simulating the attraction and repulsion forces in physics, the node distribution tends to be balanced, with closely related nodes moving closer to each other and unrelated nodes maintaining an appropriate distance.

[0113] Enhanced layered layout: The user-service (front-end service) is placed at the top layer of the topology, the order-service and product-service (business services) are placed in the middle layer, and the payment-service (payment service) and db-service (database service) are placed at the bottom layer, forming a clear call hierarchy;

[0114] Layout optimization:

[0115] Node clustering: Cluster multiple database instances (such as master and slave instances) of db-service and represent them with an aggregate node. Instance details are displayed when the mouse hovers over them.

[0116] Edge crossing optimization: Orthogonal routing algorithms are used to extend the call edges between services in the horizontal or vertical direction, reducing crossings;

[0117] Highlighting key nodes: If an instance of payment-service triggers an alarm due to a response timeout, the node of that instance will be set to red and its size will be increased to make it more prominent in the topology.

[0118] Step 6: Topology Visualization and Dynamic Updates

[0119] Visualize and render the laid-out topology, and configure dynamic updates:

[0120] Visual rendering:

[0121] Node rendering: Blue circles represent business services (such as order-service), and green rectangles represent database services (such as db-service); red nodes indicate the presence of host alarms, and orange nodes indicate application alarms.

[0122] Edge rendering: The thickness of the edge is positively correlated with the frequency of calls (the more frequent the calls, the thicker the edge). The edge color is green to indicate normal average time, yellow to indicate high time, and red to indicate excessive time. The edge is marked with "number of calls / average time", such as "300 / 60ms".

[0123] Interactive operation: Supports mouse wheel zooming of the topology, dragging nodes to adjust their positions, clicking on a node to bring up a sidebar displaying the node's detailed attributes (such as instance resource utilization, historical call trends), and clicking on a side to display the detailed link log of the call;

[0124] Dynamic update mechanism:

[0125] Scheduled updates: An update is triggered every minute to reprocess link data and update the topology;

[0126] Through the above embodiments, the method of the present invention successfully generates an intuitive topology diagram of the distributed microservice call relationship of an e-commerce platform. Technical personnel can quickly understand the call dependencies of "user service → order service → product service / payment service → database service" through this topology. At the same time, by combining the call frequency, time consumption and alarm information, problems such as "payment service response timeout" can be discovered in a timely manner, providing strong support for system operation and maintenance and optimization.

[0127] Working principle: First, probes or log collection components deployed in the application service collect real-time trace data of each service in the distributed system, including unique request identifiers, call initiator and target service identifiers, service instance information, call timestamps, time consumption, and results. Next, the raw trace data undergoes cleaning, deduplication, and format standardization preprocessing to remove invalid data and unify the data format. Then, based on the preprocessed data, application service nodes are identified and modeled, including extracting service names, associated service and instance information from the service identifier field, and defining unique identifiers, service types, alarm status, and call frequency attributes for each node. Finally, directed call edges are constructed based on the call relationships between services, determining the direction of the edges and assigning call frequency values. The topology is evaluated based on attributes such as frequency, average time consumption, and success rate. Repeated edges in the same direction are aggregated and optimized to improve readability. Then, a force-directed layout algorithm combined with a hierarchical layout approach is used to automatically arrange nodes and edges. Strategies such as node clustering, edge cross-optimization, and highlighting key nodes are further optimized to ensure a clear layout and distinct hierarchy. Finally, the optimized topology is visualized, using different shapes, colors, and thicknesses to distinguish node types, alarm statuses, and edge call frequency and time consumption, and supporting interactive operations. A timed and event-driven dynamic update mechanism is established to ensure the topology reflects system call relationships and status changes in real time, thereby achieving automated, accurate generation and visualization of distributed system application call topology.

[0128] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatically generating application call topology based on APM link data, characterized in that, Includes the following steps: Step S1: Collect link tracing data of each service in the distributed system in real time through probes or log collection components deployed in the application service. The link tracing data includes at least the unique request identifier, the identifier of the service initiating the call, the identifier of the target service, the instance information of the service initiating the call, the instance information of the target service, the call timestamp, the call time, and the call result. Step S2: Clean and preprocess the collected raw link data, including data cleaning, data deduplication, and format standardization; Step S3: Based on the preprocessed link data, identify the application service nodes in the distributed system and model each node. Step S4: Construct call edges in the topology based on the call relationships between services in the link data; Step S5: Use a graph layout algorithm to lay out the identified nodes and edges, and generate a visualized topology structure by selecting and optimizing the layout algorithm. Step S6: Visualize and render the layout topology, and establish a dynamic update mechanism with timed updates and event-driven updates.

2. The method for automatically generating application call topology based on APM link data according to claim 1, characterized in that, In step S2: Data cleaning includes: filtering invalid data; Data deduplication includes: removing duplicate link records caused by repeated reporting by the collection component based on Trace ID and call sequence; Format standardization includes converting link data from different sources into a unified structured data format to facilitate subsequent processing.

3. The method for automatically generating application call topology based on APM link data according to claim 1, characterized in that, The specific steps in step S3 of identifying application service nodes in the distributed system include: Extract all relevant service names from the Source Service and Target Service fields of the link data, and use them as service nodes in the topology; By combining service identifiers and service instance information, each service is associated with its corresponding instance node, thus clarifying the hierarchical relationship between "service and instance". The modeling of each node in step S3 includes: Define attributes for each node, including at least: unique node identifier, service name, instance information, current alarm status, service type, and historical call frequency statistics.

4. The method for automatically generating application call topology based on APM link data according to claim 1, characterized in that, The edge construction in step S4 includes: Edge direction determination: If there are call records from service A to service B in the link data, then construct a directed edge from service A to service B; Edge attribute assignment: Define attributes for each edge, including at least: call frequency, average call time, call success rate, and most recent call timestamp; Edge aggregation and optimization: For multiple call edges in the same direction and with the same source-target, aggregation processing is performed. Attributes such as call frequency and time consumption are statistically aggregated to reduce the number of edges and improve topology readability.

5. The method for automatically generating application call topology based on APM link data according to claim 1, characterized in that, The layout algorithm selection in step S5 includes: Based on the force-directed layout algorithm, it simulates the "attraction" and "repulsion" between nodes, so that the topology presents a natural and clear distribution. At the same time, combined with the concept of hierarchical layout, nodes with clear calling levels are arranged in layers to enhance the hierarchical readability of the topology. The layout optimization strategy in step S5 includes: Node clustering: Clustering and displaying multiple instance nodes belonging to the same service to avoid topology clutter caused by too many instances; Edge crossing optimization: By adjusting node positions or using edge routing algorithms, edge crossings are reduced, improving topology clarity; Highlighting key nodes: Nodes with alarm status or abnormal call frequency / time consumption are highlighted by increasing their size or using a brighter color.

6. The method for automatically generating application call topology based on APM link data according to claim 1, characterized in that, The visualization rendering in step S6 includes: Node rendering: Different shapes and colors are used to distinguish service types, and light and dark colors or flashing effects are used to indicate alarm status; Edge rendering: Different thicknesses are used to represent call frequency, different colors represent average time consumption, and key data on call frequency and average time consumption are marked on the edge; Enhanced interactivity: Supports interactive operations such as zooming and panning the topology, highlighting selected nodes, and viewing call details by selecting edges; The dynamic update mechanisms for timed updates and event-driven updates in step S6 include: Scheduled Updates: Process link data at preset time intervals, repeating steps 2-5 to update the nodes, edges, and attributes of the topology; Event-driven updates: When the APM system detects changes in service instances, call relationships, or alarm status, it triggers immediate updates to ensure that the topology reflects the system status in real time.

Images